Java Build Automation Part 1: Create lean executable jar using Maven

Have you ever tried to create an executable jar file, while keeping all configuration files outside of the jar file, accessible to operations folks? Here is a step by step guide.

JAR files are java archive files and as such, are not designed to be manipulated by operations folks („ops“). However, if a developer creates an executable jar, all needed files, including configuration and template files are packed into the jar file as a default. In this post, I will show, how to make configuration and template files accessible again.

Update/Comment (2016-11-20): I now have created a blog, which (in my opinion) shows a better way of creating a lean executable JAR file using gradle: see here.

Automate the creation of the executable jar

Up to now, I have used Eclipse’s export function to create executable jars for both, the main program and the JUnit test program.

However, this mouse-click procedure has forced me to manually extract the configuration files from the jar file, so the administrator can change the configuration files without compiling the source code again. Since I want to come a little step closer to an automated build of a Docker image for my application, I need to automate the creation of the executable jar. A short Internet research has pointed into the direction of three possible options to do so (please tell me, if you now better ones!):

1. using jar commands
2. using ant
3. using maven

About 1.: jar

jar commands are similar to tar commands and give you full flexibility to control, which folders&files are put in the jar file, choose the folder structure within the jar, and choose, which files&folders are placed outside of the jar. However, I prefer to use a higher level tool like ant or maven, since they are opinionated about the structure. This helps me to choose a structure which is close to best practice.

About 2.: ant

The ant tool has the advantage that an ant script can be created using eclipse export.

However, when looking into the resulting ant script, I am missing a possibility to control, whether of not the resources are packed into the jar file (tell me, if you know, how to do it!).

Moreover, each and every dependable jar file is explicitly copied by the ant script. In case I will add a dependency to the POM file later, the ant script needs to be adapted. This is not so optimal. In any case, I have decided to look at the next option: the maven script.

About 3.: maven

On http://www.mkyong.com/maven/how-to-create-a-jar-file-with-maven/ I have found a very good step by step documentation on how to create a executable jar. It was quite easy to apply the documentation on my Apache Camel java program. It also shows how to place the log4j properties file outside of the jar. However, I have more files than the log4j properties file, which I want to make accessible to the administrator.

Executable jar via Maven

Additions to the pom.xml file:

<!-- Make this jar runnable/executable -->
<plugin>
 <groupId>org.apache.maven.plugins</groupId>
 <!-- documentation: https://maven.apache.org/plugins/maven-jar-plugin/jar-mojo.html -->
  <artifactId>maven-jar-plugin</artifactId>
  <configuration>
  <!-- custom output directory -->
  <outputDirectory>${project.build.directory}/jarDir</outputDirectory>
  <!-- custom Name of the generated JAR (default: ${project.build.finalName} e.g. de.oveits.provisioningengine_recent-0.5.2.21_stable). Note, that .jar is always appended -->
  <finalName>OpenScapeConnector</finalName>
   <archive>
    <manifest>
     <addClasspath>true</addClasspath>
     <mainClass>de.oveits.provisioningengine.MainApp</mainClass>
     <classpathPrefix>lib/</classpathPrefix>
    </manifest>
   </archive>
  </configuration>
</plugin>

In order to avoid a „fat“ jar, i.e. a large jar, which has all dependent jars included, we use the „copy-dependencies“ plugin:

<!-- Copy project dependency -->
<plugin>
 <groupId>org.apache.maven.plugins</groupId>
 <artifactId>maven-dependency-plugin</artifactId>
 <version>2.5.1</version>
 <executions>
  <execution>
   <id>copy-dependencies</id>
   <phase>package</phase>
   <goals>
    <goal>copy-dependencies</goal>
   </goals>
   <configuration>
    <!-- exclude junit, we need runtime dependency only -->
    <includeScope>runtime</includeScope>
    <!-- custom output directory -->
    <outputDirectory>${project.build.directory}/jarDir/lib/</outputDirectory> 
  </configuration> 
</execution> 
</executions> 
</plugin>

Note: Eclipse might complain about the copy-dependencies plugin not being compatible with m2e of the eclipse IDE. In this case, you can either accept one of the offered solutions, or you can manually add the following code to the POM file:

<pluginManagement>
 <plugins>
 <!--This plugin's configuration is used to store Eclipse m2e settings only. It has no influence on the Maven build itself.-->
  <plugin>
  <groupId>org.eclipse.m2e</groupId>
  <artifactId>lifecycle-mapping</artifactId>
  <version>1.0.0</version>
  <configuration>
  <lifecycleMappingMetadata>
  <pluginExecutions>
  <pluginExecution>
  <pluginExecutionFilter>
  <groupId>
  org.apache.maven.plugins
  </groupId>
  <artifactId>
   maven-dependency-plugin
  </artifactId>
  <versionRange>
  [2.5.1,)
  </versionRange>
  <goals>
  <goal>copy-dependencies</goal>
  </goals>
  </pluginExecutionFilter>
  <action>
  <ignore></ignore>
  </action>
  </pluginExecution>
  </pluginExecutions>
  </lifecycleMappingMetadata>
  </configuration>
  </plugin>
 </plugins>
</pluginManagement>

With this, the plugin is ignored with m2e, therefore avoiding the conflict.

Then, we can create the jar file by:

mvn package -Dmaven.test.skip=true

In my case, I have skipped the JUnit tests by specifying the option „-Dmaven.test.skip=true“, because I have not implemented simulator/mock services yet and the test with the real target systems take >~30 minutes.

This step will create a jar file in the <outputDirectory> as defined in the POM file (here: target/jarDir) with the name <finalName> + jar  (here: „OpenScapeConnector.jar“).

The dependent jars are put in <outputDirectory> of the maven-dependency-plugin. The <classpathPrefix> must be the relative path from the jar file to the dependent libraries with trailing „/“ (here: lib/). Otherwise, the dependent jars will not be found.

The program then can be started with:

java -jar target/jarDir/OpenScapeConnector.jar

Here, OpenScapeConnector.jar is the <finalName> and target/jarDir is the <outputDirectory> as defined in the POM.

Note that I have kept the log4j properties in the jar file for now, which is a small deviation from the description on Mkyong’s Post. I will take care of this later below.

Extract an remove Configuration Folders from the jar File

We have created an executable jar, but this jar still contains configuration files which are not accessible easily by the operators/administrators. In my case, „cfg“ and „properties“ contain configuration files, „templates“ contains many velocity templates and the „wiretap“ folder is the destination folder for traces. All of those folders need to be easily accessible by operations folks:

I have not found any cool maven plugin that could be of any help with this task (apart from excluding the files in the jar and copy them per Linux cp commands to the target folder), so I had decided to create a jar first, and then use standard jar and zip commands to extract and delete the configuration files from the jar:

Moving out and deleting of the folders/files can be done via shell script like follows:

# extract the files and directories myFile1 myFile2 myDir1 myDir2 myDir3 from the jar:

extractList="myFile1 myFile2 myDir1 myDir2 myDir3"
jar xvf myJarFile.jar $extractList

# find and remove the files and folders from the jar. 
find $extractList -exec zip -d $jarFile {} \;

The find command recursively finds all extracted directories. The zip delete command can be used to delete the file/directory from the jar file, since jar files have the same format as zip files.

Now let us try to start the jar file:

java -jar target/jarDir/OpenScapeConnector.jar
...
Exception in thread "main" org.apache.camel.RuntimeCamelException: org.apache.camel.FailedTo CreateRouteException:...not found in classpath

The jar file does not work. What happened? Now, that we have moved some folders out of the jar file, java does not know, where to find them. All files and folders are located outside of the jar in the same directory as the jar file, so we have to add the current directory „.“ to the classpath in the Manifest file.

First, I have tried to solve that issue via maven-jar-plugin by adding

<additionalClasspathElements><additionalClasspathElement>.</additionalClasspathElement></additionalClasspathElements>

to the POM file. However, this element does not seem to work within this maven-jar-plugin: it had no effect on the resulting Manifest file (did you get it work?). Therefore, I have decided to use low level manipulations via jar, awk and zip commands instead. This is not very complex either: we need to

1. extract file from jar using jar command
2. manipulate file using awk and gsub
3. update file in jar using zip command

like follows:

#!/bin/sh
jarDir=target/jarDir
jarFile=OpenScapeConnector.jar

cd $jarDir

# 1) extract MANIFEST.MF file:
jar xvf $jarFile META-INF/MANIFEST.MF

# 2) fix MANIFEST.MF: 
awk '{gsub(/Class-Path: /, "Class-Path: . "); print $0}' META-INF/MANIFEST.MF > MANIFEST.MF.temp; mv MANIFEST.MF.temp META-INF/MANIFEST.MF 

# 3) update MANIFEST.MF in jar file: 
zip -u OpenScapeConnector.jar META-INF/MANIFEST.MF

With the jar xvf command, we have extracted the Manifest file from the jar file. With the awk command, we have added the current directory ‚.‘ as she first entry to of the Class-Path:

The zip command has updated the jar file with the manipulated Manifest file.

Now we try again to start the application:

java -jar target/jarDir/OpenScapeConnector.jar
...
6140 [main] INFO org.apache.camel.spring.SpringCamelContext - Total 94 routes, of which 94 is started.
[ main] SpringCamelContext INFO Apache Camel 2.12.2 (CamelContext: camel-1) started in 2.781 seconds
6141 [main] INFO org.apache.camel.spring.SpringCamelContext - Apache Camel 2.12.2 (CamelContext: camel-1) started in 2.781 seconds

Bingo! The java program has started with no problems.

Verification

Now let us test, that a change in one of the extracted files has the desired effect. E.g. we can change the log4j settings:

vi target/jarDir/log4j.properties

and change

log4j.logger.org.apache.camel=INFO

to

log4j.logger.org.apache.camel=DEBUG

Now at the startup of the java application, many DEBUGs are shown:

ent.event.EventComponent@5b8dbd69
[ main] DefaultManagementAgent DEBUG ...
[ main] SpringCamelContext INFO Total 94 routes, of which 94 is started.
9322 [main] INFO org.apache.camel.spring.SpringCamelContext - Total 94 routes, of which 94 is started.
[ main] SpringCamelContext INFO Apache Camel 2.12.2 (CamelContext: camel-1) started in 5.905 seconds

and the startup of the Apache Camel routes take twice as long. This is expected and shows that the administrator now can change the configuration files, and after a restart of the program, the changes are active. No re-creation of the jar file is necessary.

The same holds, if I manipulate one of the configuration files. In my web application, the IP address of the target is saved in cfg/WebPortal.cfg; i.e. one of the files, I have extracted from the jar file. Before changing the file, I see on the Web Portal page:

In order to verify that the configuration files in the cfg folder outside of the jar file are „active“, I have changed the OSVIP variable from 10.152.0.10 to 1.1.1.1 in cfg/WebPortal.cfg:

After a restart of the java application and a reload of the page, we see that the web portal content has changed:

.

The Web Portal content has been adapted without the need to re-compile the jar file: Bingo! Our goal has been achieved.

Summary

We have shown how to automate the creation of an executable jar file using maven. In order to create a „lean“ jar, the dependent jars were extracted and placed outside the jar using a maven plugin. In order to give administrators the chance to change configuration files without re-compiling the jar from source, we have extracted those files using jar and zip commands. Finally, the search path (classpath) had to be manipulated within the jar file, using jar, awk and zip commands.

Note that I am still left with the manual task of the test JUnit jar creation. The maven-jar-plugin does not seem to be designed to be usable for both, the main application and its corresponding JUnit test application, see https://maven.apache.org/plugins/maven-jar-plugin/examples/create-test-jar.html for details. The „easy way“ described there did not work for me („cannot find main class“, when I try to start the jar). So, either I need to find, what I am doing wrong, or I need to follow the „preferred way“, i.e. to move the test code to a separate project with its own POM file and apply the same procedure to the test main app as I did for the main application.