Red Hat Fuse (JBoss Fuse) and Fuse Fabric

Quickstarts

Fuse 7.8 on OpenShift quick demo

A super-quick, quickstart to create and deploy a Fuse application on OpenShift. This assumes you’re a developer without cluster-admin access:

1. Create a new application from the Fuse Maven archetype, e.g.:

   For Fuse 7.8 / Spring Boot 2.x:

   mvn org.apache.maven.plugins:maven-archetype-plugin:2.4:generate \
   -DarchetypeCatalog=https://maven.repository.redhat.com/ga/io/fabric8/archetypes/archetypes-catalog/2.2.0.fuse-sb2-780040-redhat-00001/archetypes-catalog-2.2.0.fuse-sb2-780040-redhat-00001-archetype-catalog.xml \
   -DarchetypeGroupId=org.jboss.fuse.fis.archetypes \
   -DarchetypeArtifactId=spring-boot-camel-config-archetype \
   -DarchetypeVersion=2.2.0.fuse-sb2-780040-redhat-00001 \
   -DgroupId=foo \
   -DartifactId=bar \
   -Dpackage=com.foo.bar
   

2. Create a secret containing your Red Hat credentials - this will allow OpenShift to pull the Fuse image from the registry:

   export RH_USERNAME=
   export RH_PASSWORD=
   oc create secret docker-registry redhat-secret --docker-username=${RH_USERNAME} --docker-password=${RH_PASSWORD} --docker-server=registry.redhat.io
   

3. Install the Fuse imagestreams to your OpenShift cluster. This assumes you don’t have cluster-admin permissions, and will just install the image streams into your local namespace:

   BASEURL=https://raw.githubusercontent.com/jboss-fuse/application-templates/application-templates-2.1.fuse-750056-redhat-00004
   oc create [-n openshift] -f ${BASEURL}/fis-image-streams.json
   

4. Log in to OpenShift, build and deploy the application:

   oc login -u ...
   oc new-project myproject
   mvn clean deploy -Popenshift -Dfabric8.generator.from=$(oc project -q)/fuse7-java-openshift:1.5
   

BOOM! 💣 🦕

Component versions

Fuse standalone

The JBoss Fuse BOM (Bill of Materials) is a parent POM that defines the versions for all of the Maven artifacts provided by JBoss Fuse

• JBoss Fuse BOMs are located in the Red Hat GA Maven repository.

Here is a table of Fuse releases and the corresponding component versions (not guaranteed to be 100% correct!):

And for Fuse 7.0:

The BOM for Spring Boot projects should be referenced in a Maven POM like this (from Fuse 7.0 it’s org.jboss.redhat-fuse):

  <dependencyManagement>
    <dependencies>
      <dependency>
        <groupId>org.jboss.redhat-fuse</groupId>
        <artifactId>fuse-springboot-bom</artifactId>
        <version>${bom.version}</version>
        <type>pom</type>
        <scope>import</scope>
      </dependency>
    </dependencies>
  </dependencyManagement>

For older Fuse projects, the BOM is under org.jboss.fuse.bom:

<project ...>
  ...
  <dependencyManagement>
    <dependencies>
      <dependency>
        <groupId>org.jboss.fuse.bom</groupId>
        <artifactId>jboss-fuse-parent</artifactId>
        <version>${jboss.fuse.bom.version}</version>
        <type>pom</type>
        <scope>import</scope>
      </dependency>
    </dependencies>
  </dependencyManagement>
  ...
</project>

Fuse Integration Services 2.0 (Fuse on OpenShift)

The FIS BOM should be referenced in a Maven POM like this:

<project ...>
    ...
    <dependencyManagement>
        <dependencies>
            <dependency>
                <groupId>io.fabric8</groupId>
                <artifactId>fabric8-project-bom-camel-spring-boot</artifactId>
                <version>${fabric8.version}</version>
                <type>pom</type>
                <scope>import</scope>
            </dependency>
        </dependencies>
    </dependencyManagement>
    ...
</project>

Fuse 7

Including Fuse on OpenShift (formerly known as Fuse Integration Services).

Distributions

Fuse is available as:

• Fuse on Apache Karaf (standalone)
• Fuse on Apache Karaf (containerised - for OpenShift)
• Fuse on Spring Boot
• Fuse on Wildfly/EAP

Fuse on Spring Boot

Image streams for OpenShift

To install the Fuse image streams:

BASEURL=https://raw.githubusercontent.com/jboss-fuse/application-templates/application-templates-2.1.fuse-740025-redhat-00003
oc create -n openshift -f ${BASEURL}/fis-image-streams.json

Fuse on Apache Karaf

The standalone (non-containerised) distribution of JBoss Fuse uses Apache Karaf as its runtime:

Apache Karaf is a generic platform providing higher level features and services specifically designed for creating OSGi-based servers.

JBoss Fuse uses Apache Felix as the OSGi framework (this is configured in etc/config.properties)

Red Hat Fuse default web port

The default port for web applications (e.g. CXF REST/SOAP services) is 8181, as set in org.ops4j.pax.web.cfg:

cd $FUSE_HOME
grep 'http.port' etc/org.ops4j.pax.web.cfg

Using Hot-deploy (Felix file installer)

Hot-deploy allows components to be dropped into a folder, automatically discovered and started. To debug this behaviour, set a logger in org.ops4j.pax.logging.cfg:

log4j2.logger.felix.name = org.apache.felix.fileinstall
log4j2.logger.felix.level = DEBUG

Working with bundles in Fuse

• The osgi:list command doesn’t show all bundles by default - use the -t command to set the start-level threshold to show all bundles, e.g. osgi:list -t 0

How to make sure you’re using the correct Red Hat builds of bundles?

• Look at the file org.apache.karaf.features.xml
• This contains an element bundleReplacements which tells Karaf, when asked to install bundles at certain versions, to replace with the Red Hat-specific version of that bundle.

Running JBoss Fuse with jenv

Running Fuse 6.0 on Mac OS X (with jenv installed to manage JRE versions):

cd $FUSE_HOME
jenv local 1.7
jenv exec ./bin/fuse

Fuse on Karaf on OpenShift

Binary builds

To do a binary build for the Fuse on Karaf image, make sure that you’ve built a Karaf assembly (.zip) using the archive goal of the karaf-maven-plugin.

Then, in your S2I BuildConfig, make sure that the environment variable ARTIFACT_DIR is set to .:

strategy:
  sourceStrategy:
    env:
      - name: ARTIFACT_DIR
        value: .
    from:
      kind: ImageStreamTag
      name: fuse7-karaf-openshift:1.2
      namespace: openshift
  type: Source

Then, you can start the build like the command below, which will upload your Karaf assembly zip, and unpack it into the container image:

oc start-build my-karaf-app --from-file=target/myapp-1.0.zip --follow

Fuse 6 to 7 migration

Things to be aware of when migrating Fuse 6 to 7:

Fuse 6.x Fabric

Fabric is a Fuse feature that provides centralised configuration and bundle provisioning.

Fabric terminology 101:

• Ensemble - “a collection of Fabric Servers that collectively maintain the state of the fabric registry”
• Fabric Server - “responsible for maintaining a replica of the fabric registry”
• Fabric Container/managed container - where the actual bundles are deployed/run; “can retrieve registry data from any Fabric Server in the Fabric Ensemble”
• Root container - this is the first container when creating a Fabric; NB: it is not a Fabric-managed container, so it cannot be started/stopped using container-start, container-stop commands.

Other good things to know about Fabric:

• Fabric managed containers don’t use Karaf’s features service. Provisioning happens exclusively via Fabric profiles. So features: commands won’t be visible, unless you inherit from the jboss-fuse-minimal profile, which adds the commands to the container.

Quick Fabric setup/demo

Create a quick Fabric and deploy an example feature:

fabric:create --wait-for-provisioning
fabric:container-create-child root fabric001
fabric:profile-create --parent default foo
fabric:container-change-profile fabric001 foo

fabric:profile-edit --repository mvn:org.apache.camel/camel-example-osgi/2.17.0/xml/features foo
fabric:profile-edit --feature camel-example-osgi foo

fabric:profile-edit --delete --feature camel-example-osgi foo

Advanced/recommended Fabric creation

Creating a Fabric with more appropriate settings for a ‘real’ environment:

fabric:create ... --zookeeper-purge-interval 24 \
    --zookeeper-snap-retain-count 3

Starting/stopping containers

To start Managed Containers or Fabric Servers (except the root container):

fabric:container-start containername01
fabric:container-stop containername01

To start/stop the root container, use the start/stop scripts in the Fuse directory, or a system service wrapper, if you have configured one.

Viewing container info

How to view container info, including things like HTTP URLs (i.e. which port Jetty is running on), Jolokia URL, JMX URL, etc.:

> fabric:container-info containername01
...
SSH Url:        yourcontainer.example.com:21000
JMX Url:        service:jmx:rmi://yourcontainer.example.....
Http Url:       http://yourcontainer.exampe.com:21001
...etc...

Inspecting profile config

Profiles are stored in Git inside $KARAF_HOME/data/git/local/fabric/fabric.

Deleting a Fabric (not recommended!)

To delete a Fabric, shut down all containers then delete data and instances (NB: everything will be lost!):

rm -rf data/
rm -rf instances/

Fabric versioning

• When creating a new version (fabric:version-create), Fabric sorts version names based on the numeric version string
• To add a text description to a version name, append it to the end of the version number, e.g. 1.0.myversion

Maven Proxy and Fabric Agent

The Maven Proxy runs on each Fabric server (ensemble member) and is a central cache of Maven artifacts for the Fabric containers.

• “Managed containers try to download from the Maven proxy, before trying to download from the Internet”
• Maven Proxies are configured in a master-slave cluster on the Fabric servers
• Query Zookeeper to find out the current master: cluster-list servlets/io.fabric8.fabric-maven-proxy
• Artifacts are not automatically replicated between different Maven proxies in the cluster

The Fabric Agent runs on each managed container:

• The agent “provisions the container according to the profiles assigned to it… It retrieves any required Maven artifacts from the Maven repositories specified by its profile, which are accessed through the Maven proxies managed by the fabric.”

OSGi property reference (change properties in the default profile to ensure they are propagated across all containers in the Fabric):

Clustered AMQ

Fabric can be used to create clusters of ActiveMQ brokers.

Create a broker using:

fabric:mq-create mybroker
# OR
fabric:mq-create --group mybrokers broker1

MQ brokers have a Fabric profile created that follows the naming convention x-y-z-BrokerGroup.BrokerName, e.g. mq-broker-central.broker1

List the clusters managed by Fabric using:

fabric:cluster-list amq

For brokers managed by Fabric, broker properties should be added to a PID named io.fabric8.mq.fabric.server-$brokerName (see MQService.java), for example:

kind = MasterSlave
connectors = OpenWire
ssl = false
group = MyBrokerGroup    # this is the cluster name visible in fabric:cluster-list
                         # clients using discovery should connect to a group
broker-name = my-broker

When creating Fabric brokers manually, using profile configuration and specific ports (i.e. when not using MQ discovery), roughly follow these steps:

1. Create a parent Fabric profile for the brokers first, e.g. mq-broker-mybrokerprofile - inheriting mq-base
2. Set configuration in the PID io.fabric8.mq.fabric.server-mybrokername
3. Add the activemq.xml as a resource (file) into the profile
4. Use child profiles to override the port value for each broker in the cluster

For example: a parent profile mybrokers could inherit mq-base, with child profiles mybrokers.broker1 and mybrokers.broker2.

Patching JBoss Fuse on Karaf

To install a rollup patch into a brand new environment, just extract the rollup patch ZIP file:

Since JBoss Fuse 6.2.1, a rollup patch file is a complete new build of the official target distribution. In other words, it is just like the original GA distribution, except that it includes later build artifacts.

– Configuring and Running JBoss Fuse

To patch an existing JBoss Fuse installation, you will need the following from the Red Hat Customer Portal:

• The latest JBoss Fuse GA distribution (e.g. jboss-fuse-karaf-6.3.0.redhat-xxx.zip).
• (Optional) A Patch Mechanism upgrade distribution (Red Hat JBoss Fuse 6.x Rollup N Patch Management Package - e.g. patch-management-package-6.3.0.redhat-yyy.zip) (if applicable to this Fuse version)
• Latest Rollup Patch distribution (Red Hat JBoss Fuse 6.x Rollup N on Karaf - e.g. jboss-fuse-karaf-6.3.0.redhat-yyy.zip)

To install a hotfix patch, follow the instructions in the Fabric Guide on applying an incremental patch.

Patching steps

1. Provision a standalone Fuse container, using the GA release.
2. Copy the Patch Mechanism upgrade distribution to the remote server and follow the steps in Configuring and Running JBoss Fuse - Chapter 19. Applying Patches.
3. Copy the Rollup Patch distribution to the remote server and follow the instructions for applying a Rollup Patch, which can be found in the same guide.

For Fuse Patches, the container may need a restart. If it does, it will restart automatically. But you won’t get a warning about this.

Cookbook

Developer cookbook

Creating a new project

JBoss Fuse

Archetypes are labelled under io.fabric8.archetypes, version 1.2.0.redhat-xxxxxx. To create a new JBoss Fuse “router” project:

mvn archetype:generate -DarchetypeGroupId=io.fabric8.archetypes \
    -DarchetypeVersion=1.2.0.redhat-630187 \
    -DarchetypeArtifactId=karaf-camel-cbr-archetype \
    -DgroupId=org.fusesource.example -DartifactId=camel-basic \
    -Dversion=1.0-SNAPSHOT -Dfabric8-profile=camel-basic-profile

Or substitute with one of the following archetypes depending on the use case:

Fuse on OpenShift

To create a Fuse 7.4 on OpenShift application using the Maven archetype:

mvn org.apache.maven.plugins:maven-archetype-plugin:2.4:generate \
  -DarchetypeCatalog=https://maven.repository.redhat.com/ga/io/fabric8/archetypes/archetypes-catalog/2.2.0.fuse-740017-redhat-00003/archetypes-catalog-2.2.0.fuse-740017-redhat-00003-archetype-catalog.xml \
  -DarchetypeGroupId=org.jboss.fuse.fis.archetypes \
  -DarchetypeArtifactId=spring-boot-camel-xml-archetype \
  -DarchetypeVersion=2.2.0.fuse-740017-redhat-00003

See the Component Versions section at the top of the page for a table showing mappings of build versions Fuse releases.

Good practice

• Use the correct JBoss Fuse parent BOM for the version of Fuse that you are looking to target.

Persistence

• The Apache OpenJPA implementation of the Java Persistence API (JPA) is deprecated since 6.2.1. It is recommended to use the Hibernate implementation instead.

Connecting to ActiveMQ

A simple example (note the declaration of init-method and destroy-method):

<bean id="internalConnectionFactory" class="org.apache.activemq.ConnectionFactory">
  <argument value="tcp://localhost:61616" />
</bean>
<bean id="connectionFactory" class="org.apache.activemq.jms.pool.PooledConnectionFactory" init-method="start" destroy-method="stop">
  <property name="connectionFactory" ref="internalConnectionFactory"/>
  <property name="name" value="activemq" />
  <property name="maxConnections" value="2" />
  <property name="blockIfSessionPoolIsFull" value="true" />
</bean>

Fuse 6.x Karaf command line

Working with bundles and features

Install a bundle in Apache Karaf:

osgi:install -s mvn:groupId/artifactId/version

List all bundles with a start-level of 0 or higher, and show their location (e.g. Maven co-ordinates of the bundle):

osgi:list -t 0 -l

Install a feature:

features:addurl mvn:org.apache.camel/camel-example-osgi/2.17.0/xml/features
features:install camel-example-osgi

Show OSGi manifest information about a bundle (shows Export-Service, Export-Package, Import-Package, etc.):

osgi:headers camel-core

Show information about a feature:

features:info jasypt-encryption
features:info -t jasypt-encryption  # Shows feature tree

Debugging Karaf with IntelliJ

Find the specific JAR you want to debug and add it to the External Libraries list in IntelliJ. Then add a Run Configuration for debugging (see the IntelliJ page). Then, run Fuse like this:

export KARAF_DEBUG=true
export JAVA_DEBUG_OPTS=-agentlib:jdwp=transport=dt_socket,server=y,suspend=y,address=5005
./bin/fuse

OSGi Config Admin

View details of a specific OSGi Config Admin PID:

config:list "(service.pid=com.example.my.pid.name)"

Jolokia

Get the root info from Jolokia:

curl -vk -u admin:admin http://localhost:8181/jolokia/list

Get ActiveMQ broker info:

curl http://quarkus:quarkus@localhost:8161/console/jolokia/read/org.apache.activemq.artemis:broker\=\*

User administration

Get the list of JAAS realms:

jaas:realms

Taking the index value of the realm you want to edit from the jaas:realms list above (usually the karaf realm), use the following to create a user and add it into the manager and viewer groups:

jaas:manage --module io.fabric8.jaas.ZookeeperLoginModule --realm karaf
jaas:useradd jsmith p455w0rd
jaas:groupadd jsmith manager
jaas:groupadd jsmith viewer
jaas:update

To update an existing user’s password, use useradd against the realm:

jaas:useradd username newpassword

NB: If you update the password of the admin user, you will need to re-start your Fuse client session. (This is because it will still attempt to use the old password). Also, changing the admin password will not also update the Fabric Ensemble password. (You can verify this using fabric:ensemble-password) Details on how to update the Ensemble password are below.

Fabric

Add a URL to a feature repository to a Fabric profile:

fabric:profile-edit --repository mvn:groupId/artifactId/version/xml/features my-profile-name

Show details of a profile (shows all repositories, features, etc):

fabric:profile-display foo

Modify a config property:

fabric:profile-edit --config MY_VAR=myvalue my-profile-name

Modify a PID value in a Fabric profile:

fabric:profile-edit --pid "org.mypackage/org.hello.valueName=1234" my-profile-name

Increase global logging to DEBUG level (warning: very chatty!):

fabric:profile-edit --pid "org.ops4j.pax.logging/log4j.rootLogger=DEBUG, out, osgi:*" karaf

Change log file (fuse.log) max size and number of backups/rotation in Fabric:

fabric:profile-edit --pid org.ops4j.pax.logging/log4j.appender.out.maxFileSize=100MB karaf
fabric:profile-edit --pid org.ops4j.pax.logging/log4j.appender.out.maxBackupIndex=666 karaf

Set log levels for a specific Fabric profile:

fabric:profile-edit --pid org.ops4j.pax.logging/log4j.category.org.example.mypackage=DEBUG my-profile-name

Change the Maven repositories searched by the Fabric Agent:

fabric:profile-edit --pid io.fabric8.agent/org.ops4j.pax.url.mvn.repositories=http://nexus.example.com:8081/blah default

Update Zookeeper password:

fabric:ensemble-password newpassword123
// wait a few moments while the new password is propagated
fabric:ensemble-password --commit

Zookeeper (for Fabric)

Install Zookeeper commands and query Zookeeper (you will need to add the feature to an appropriate profile - e.g. fabric-ensemble-0000-1 or default):

fabric:profile-edit --feature fabric-zookeeper-commands <profile-name>

zk:list -r -d /fabric/registry/path
zk:get /fabric/registry/path/mykey

Some useful locations of things in Zookeeper:

Other commands and one-liners

Connect to a remote Fuse (Karaf) instance:

$ ./bin/client -a 8161 -h fuseserver.company.com -u admin

Extract the current Provision Status of a Fabric container, using container-info and awk:

$ ./bin/client -u admin -p admin fabric:container-info my-container-name | grep 'Provision Status' | tr -s ' ' | awk -F": " '{print $2}'
success

Troubleshooting

Clear the Karaf cache (where bundles are stored):

rm -rf data/cache

Bundles get stuck in GracePeriod status and then fail:

• Most often, the failure of a bundle is due to a missing dependency. Common reasons:
  • A Karaf feature has not been installed (view all installed features using features:list). This might have neglected to be done manually, or perhaps the Fabric profile being used does not install or inherit the required feature.
  • A datasource does not exist, or is configured incorrectly.
  • An ActiveMQ connection is pointing at the wrong host/port (look for “Failed to connect to […] after: 10 attempt(s) continuing to retry.”)
• Check the logs (log:display) for lines like Bundle my-bundle/1.0.0-SNAPSHOT is waiting for dependencies [(objectClass=com.example.MyClass), (objectClass=com.example.AnotherClass)] - this shows the classes that a bundle is waiting for, so that it can start.
  • Then look for the bundles which export those dependent classes. If those bundles fail to start, this will cause other bundles to wait and then eventually fail - use osgi:find-class com.example.MyClass to see which bundles contain a given class.
• Check the logs for “Application context refresh failed” - this may give some clues, if a bean declared in the bundle’s blueprint.xml is configured incorrectly or cannot be instantiated.
• If using Hibernate, check the logs for “Error creating EntityManagerFactory” (can be seen as a WARN level log). This should be accompanied by a stack trace, which will show the reason why the EntityManager couldn’t be created - often the result of an invalid JPA mapping.

Bundles stuck at Resolved status:

• Check that the startlevel property in config.properties is high enough to start all bundles. If bundles are assigned a start-level which is higher than the startlevel property, they will remain at Resolved and not move to Active.

Fabric agent does not notice updated SNAPSHOT numbers, and continues to deploy old snapshot versions of bundles to Fabric containers:

• Remove the Fabric profile from containers (container-remove-profile ...), delete the profile (profile-delete) and then recreate it.
• (Generally this should only be a problem in your development environment)

Fabric Agent tries to resolve Fuse dependencies from online repositories (Maven Central, etc.) rather than through the local system repository of the Fabric Maven Proxy:

• Check that there is no web proxy configuration required to communicate with the Fabric Maven Proxy

javax.management.InstanceAlreadyExistsException occurs when starting Fuse:

• This happens when the installation directory is moved/copied to a new location

Bundles are waiting for EntityManagerFactory (error log seen is Unable to start blueprint container for bundle ... due to unresolved dependencies (...osgi.unit.name=MyPersistenceUnit)(objectClass=javax.persistence.EntityManagerFactory)):

• Check that the datasource bundle is exporting the data sources required, using osgi:ls <bundle-id>
• Look for any instances of org.hibernate.MappingException which may indicate that JPA mapping annotations are incorrect - e.g. repeated columns

Felix Fileinstall doesn’t seem to pick up Blueprint XMLs / Karaf just ignores XML files in the hot-deploy folder:

• Check that the deployer feature is installed into Karaf. This provides the bundle org.apache.karaf.deployer.blueprint, amongst others.
• If there is not a suitable deployer present which can handle a file (e.g. a Blueprint XML file), Felix Fileinstall will just skip over it.

karaf-maven-plugin fails to build a Karaf microcontainer; build fails with error “null”:

• Check that all KAR archives have been downloaded successfully from Maven Central (or another mirror).
• The karaf-maven-plugin builds a Karaf microcontainer by downloading KAR archive(s) from a Maven repository.
• If the KAR file is corrupted, or hasn’t been downloaded properly, you might not see an error – instead, the process might just fail silently at some later stage, without giving a clue that the KAR was invalid.

karaf-maven-plugin builds a Karaf container without the commands/features that you want:

• The logs from the assembly build should say something like: “adding all non-blacklisted features from repository: mvn:org.jboss.fuse/fuse-karaf-framework/7.10.0.fuse-7_10_2…/xml/features”
• Open this Maven artifact (it’s an XML file) to see the details of the features, and which bundles they contain. This Maven artifact also pulls in other XML feature files.
• The feature called bundle is defined in org.apache.karaf.features/standard/4.2.12…/xml/features. It installs the bundle org.apache.karaf.bundle.core, which provides Karaf CLI commands like list