Spring Boot and multiple external configuration files

I have multiple property files that I want to load from classpath. There is one default set under `/src/main/resources` which is part of `myapp.jar`. My `springcontext` expects files to be on the classpath. i.e.

<util:properties id="Job1Props"
location="classpath:job1.properties"></util:properties>

<util:properties id="Job2Props"
location="classpath:job2.properties"></util:properties>

I also need the option to override these properties with an external set. I have an external config folder in `cwd`. As per spring boot doc config folder should be on classpath. But its not clear from doc if it will only override the `application.properties` from there or all the properties in config.

When I tested it, only `application.properties` gets picked up and rest of properties are still picked up from `/src/main/resources`. I have tried supplying them as comma separated list to `spring.config.location` but the default set is still not being overriden.

How do I make multiple external config files override default ones?

As workaround I currently used `app.config.location` (app specific property) which I supply through the command line. i.e

java -jar myapp.jar app.config.location=file:./config

and I changed my `applicationcontext` to

<util:properties id="Job1Props"
location="{app.config.location}/job1.properties"></util:properties>

<util:properties id="Job2Props"
location="{app.config.location}/job2.properties"></util:properties>

And this is how I make separation between file and classpath while loading Application. <br/>
**EDITS:**

//pseudo code

if (StringUtils.isBlank(app.config.location)) {
System.setProperty(APP_CONFIG_LOCATION, "classpath:");
}

I would really like not to use the above workaround and have Spring override all external config files on the classpath like it does for the `application.properties` file.

Take a look at the PropertyPlaceholderConfigurer, I find it clearer to use than annotation.

e.g.



@Configuration
public class PropertiesConfiguration {


@Bean
public PropertyPlaceholderConfigurer properties() {
final PropertyPlaceholderConfigurer ppc = new PropertyPlaceholderConfigurer();
// ppc.setIgnoreUnresolvablePlaceholders(true);
ppc.setIgnoreResourceNotFound(true);

final List<Resource> resourceLst = new ArrayList<Resource>();

resourceLst.add(new ClassPathResource("myapp_base.properties"));
resourceLst.add(new FileSystemResource("/etc/myapp/overriding.propertie"));
resourceLst.add(new ClassPathResource("myapp_test.properties"));
resourceLst.add(new ClassPathResource("myapp_developer_overrides.properties")); // for Developer debugging.

ppc.setLocations(resourceLst.toArray(new Resource[]{}));

return ppc;
}

I've just had a similar problem to this and finally figured out the cause: the application.properties file had the wrong ownership and rwx attributes. So when tomcat started up the application.properties file was in the right location, but owned by another user:

$ chmod 766 application.properties

$ chown tomcat application.properties

I had the same problem. I wanted to have the ability to overwrite an internal configuration file at startup with an external file, similar to the Spring Boot application.properties detection.
In my case it's a user.properties file where my applications users are stored.

My requirements:

Load the file from the following locations (in this order)

1. The classpath
2. A */config* subdir of the current directory.
3. The current directory
4. From directory or a file location given by a command line parameter at startup

I came up with the following solution:

import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
import org.springframework.beans.factory.annotation.Value;
import org.springframework.context.annotation.Bean;
import org.springframework.core.io.ClassPathResource;
import org.springframework.core.io.PathResource;
import org.springframework.core.io.Resource;

import java.io.IOException;
import java.util.Properties;

import static java.util.Arrays.stream;

@Configuration
public class PropertiesConfig {

private static final Logger LOG = LoggerFactory.getLogger(PropertiesConfig.class);

private final static String PROPERTIES_FILENAME = "user.properties";

@Value("${properties.location:}")
private String propertiesLocation;

@Bean
Properties userProperties() throws IOException {
final Resource[] possiblePropertiesResources = {
new ClassPathResource(PROPERTIES_FILENAME),
new PathResource("config/" + PROPERTIES_FILENAME),
new PathResource(PROPERTIES_FILENAME),
new PathResource(getCustomPath())
};
// Find the last existing properties location to emulate spring boot application.properties discovery
final Resource propertiesResource = stream(possiblePropertiesResources)
.filter(Resource::exists)
.reduce((previous, current) -> current)
.get();
final Properties userProperties = new Properties();

userProperties.load(propertiesResource.getInputStream());

LOG.info("Using {} as user resource", propertiesResource);

return userProperties;
}

private String getCustomPath() {
return propertiesLocation.endsWith(".properties") ? propertiesLocation : propertiesLocation + PROPERTIES_FILENAME;
}

}

Now the application uses the classpath resource, but checks for a resource at the other given locations too. The last resource which exists will be picked and used.
I'm able to start my app with java -jar myapp.jar --properties.location=/directory/myproperties.properties to use an properties location which floats my boat.

An important detail here: Use an empty String as default value for the properties.location in the @Value annotation to avoid errors when the property is not set.

The convention for a properties.location is: Use a directory or a path to a properties file as properties.location.

If you want to override only specific properties, a PropertiesFactoryBean with setIgnoreResourceNotFound(true) can be used with the resource array set as locations.

I'm sure that this solution can be extended to handle multiple files...

**EDIT**

Here my solution for multiple files :) Like before, this can be combined with a PropertiesFactoryBean.

import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
import org.springframework.beans.factory.annotation.Value;
import org.springframework.context.annotation.Bean;
import org.springframework.context.annotation.Configuration;
import org.springframework.core.io.ClassPathResource;
import org.springframework.core.io.PathResource;
import org.springframework.core.io.Resource;

import java.io.IOException;
import java.util.Map;
import java.util.Properties;

import static java.util.Arrays.stream;
import static java.util.stream.Collectors.toMap;

@Configuration
class PropertiesConfig {

private final static Logger LOG = LoggerFactory.getLogger(PropertiesConfig.class);
private final static String[] PROPERTIES_FILENAMES = {"job1.properties", "job2.properties", "job3.properties"};

@Value("${properties.location:}")
private String propertiesLocation;

@Bean
Map<String, Properties> myProperties() {
return stream(PROPERTIES_FILENAMES)
.collect(toMap(filename -> filename, this::loadProperties));
}

private Properties loadProperties(final String filename) {
final Resource[] possiblePropertiesResources = {
new ClassPathResource(filename),
new PathResource("config/" + filename),
new PathResource(filename),
new PathResource(getCustomPath(filename))
};
final Resource resource = stream(possiblePropertiesResources)
.filter(Resource::exists)
.reduce((previous, current) -> current)
.get();
final Properties properties = new Properties();

try {
properties.load(resource.getInputStream());
} catch(final IOException exception) {
throw new RuntimeException(exception);
}

LOG.info("Using {} as user resource", resource);

return properties;
}

private String getCustomPath(final String filename) {
return propertiesLocation.endsWith(".properties") ? propertiesLocation : propertiesLocation + filename;
}

}

With Spring boot , the spring.config.location does work,just provide comma separated properties files.

see the below code

@PropertySource(ignoreResourceNotFound=true,value="classpath:jdbc-${spring.profiles.active}.properties")
public class DBConfig{

@Value("${jdbc.host}")
private String jdbcHostName;
}
}

one can put the default version of jdbc.properties inside application. The external versions can be set lie this.

java -jar target/myapp.jar --spring.config.location=classpath:file:///C:/Apps/springtest/jdbc.properties,classpath:file:///C:/Apps/springtest/jdbc-dev.properties

Based on profile value set using spring.profiles.active property, the value of jdbc.host will be picked up.
So when (on windows)

set spring.profiles.active=dev

jdbc.host will take value from jdbc-dev.properties.

for

set spring.profiles.active=default


jdbc.host will take value from jdbc.properties.

this is one simple approach using spring boot

**TestClass.java**

@Configuration
@Profile("one")
@PropertySource("file:/{selected location}/app.properties")
public class TestClass {

@Autowired
Environment env;

@Bean
public boolean test() {
System.out.println(env.getProperty("test.one"));
return true;
}
}

the **app.properties** context, in your *selected location*

test.one = 1234

your **spring boot application**

@SpringBootApplication

public class TestApplication {

public static void main(String[] args) {
SpringApplication.run(testApplication.class, args);
}
}
and the predefined **application.properties** context

spring.profiles.active = one

you can write as many configuration class as you like and enable/disable them just by setting *spring.profiles.active* = the profile name/names {separated by commas}

as you can see spring boot is great it just needs sometime to get familiar with, it's worth mentioning that you may use @Value on your fields as well

@Value("${test.one}")
String str;

spring boot allows us to write different profiles to write for different environments, for example we can have separate properties files for production, qa and local environments

application-local.properties file with configurations according to my local machine is

spring.profiles.active=local

spring.data.mongodb.host=localhost
spring.data.mongodb.port=27017
spring.data.mongodb.database=users
spring.data.mongodb.username=humble_freak
spring.data.mongodb.password=freakone

spring.rabbitmq.host=localhost
spring.rabbitmq.username=guest
spring.rabbitmq.password=guest
spring.rabbitmq.port=5672

rabbitmq.publish=true

Similarly, we can write application-prod.properties and application-qa.properties as many properties files as we want

then write some scripts to start the application for different environments, for e.g.

mvn spring-boot:run -Drun.profiles=local
mvn spring-boot:run -Drun.profiles=qa
mvn spring-boot:run -Drun.profiles=prod

I have found this to be a useful pattern to follow:

@RunWith(SpringRunner)
@SpringBootTest(classes = [ TestConfiguration, MyApplication ],
properties = [
"spring.config.name=application-MyTest_LowerImportance,application-MyTest_MostImportant"
,"debug=true", "trace=true"
]
)

Here we override the use of "application.yml" to use "application-MyTest_LowerImportance.yml" and also "application-MyTest_MostImportant.yml"
(Spring will also look for .properties files)

Also included as an extra bonus are the debug and trace settings, on a separate line so you can comment them out if required ;]

The debug/trace are incredibly useful as Spring will dump the names of all the files it loads and those it tries to load.
You will see lines like this in the console at runtime:

TRACE 93941 --- [ main] o.s.b.c.c.ConfigFileApplicationListener : Skipped config file 'file:./config/application-MyTest_MostImportant.properties' (file:./config/application-MyTest_MostImportant.properties) resource not found
TRACE 93941 --- [ main] o.s.b.c.c.ConfigFileApplicationListener : Skipped config file 'file:./config/application-MyTest_MostImportant.xml' (file:./config/application-MyTest_MostImportant.xml) resource not found
TRACE 93941 --- [ main] o.s.b.c.c.ConfigFileApplicationListener : Skipped config file 'file:./config/application-MyTest_MostImportant.yml' (file:./config/application-MyTest_MostImportant.yml) resource not found
TRACE 93941 --- [ main] o.s.b.c.c.ConfigFileApplicationListener : Skipped config file 'file:./config/application-MyTest_MostImportant.yaml' (file:./config/application-MyTest_MostImportant.yaml) resource not found
TRACE 93941 --- [ main] o.s.b.c.c.ConfigFileApplicationListener : Skipped config file 'file:./config/application-MyTest_LowerImportance.properties' (file:./config/application-MyTest_LowerImportance.properties) resource not found
TRACE 93941 --- [ main] o.s.b.c.c.ConfigFileApplicationListener : Skipped config file 'file:./config/application-MyTest_LowerImportance.xml' (file:./config/application-MyTest_LowerImportance.xml) resource not found
TRACE 93941 --- [ main] o.s.b.c.c.ConfigFileApplicationListener : Skipped config file 'file:./config/application-MyTest_LowerImportance.yml' (file:./config/application-MyTest_LowerImportance.yml) resource not found
TRACE 93941 --- [ main] o.s.b.c.c.ConfigFileApplicationListener : Skipped config file 'file:./config/application-MyTest_LowerImportance.yaml' (file:./config/application-MyTest_LowerImportance.yaml) resource not found
TRACE 93941 --- [ main] o.s.b.c.c.ConfigFileApplicationListener : Skipped config file 'file:./application-MyTest_MostImportant.properties' (file:./application-MyTest_MostImportant.properties) resource not found
TRACE 93941 --- [ main] o.s.b.c.c.ConfigFileApplicationListener : Skipped config file 'file:./application-MyTest_MostImportant.xml' (file:./application-MyTest_MostImportant.xml) resource not found
TRACE 93941 --- [ main] o.s.b.c.c.ConfigFileApplicationListener : Skipped config file 'file:./application-MyTest_MostImportant.yml' (file:./application-MyTest_MostImportant.yml) resource not found
TRACE 93941 --- [ main] o.s.b.c.c.ConfigFileApplicationListener : Skipped config file 'file:./application-MyTest_MostImportant.yaml' (file:./application-MyTest_MostImportant.yaml) resource not found
TRACE 93941 --- [ main] o.s.b.c.c.ConfigFileApplicationListener : Skipped config file 'file:./application-MyTest_LowerImportance.properties' (file:./application-MyTest_LowerImportance.properties) resource not found
TRACE 93941 --- [ main] o.s.b.c.c.ConfigFileApplicationListener : Skipped config file 'file:./application-MyTest_LowerImportance.xml' (file:./application-MyTest_LowerImportance.xml) resource not found
TRACE 93941 --- [ main] o.s.b.c.c.ConfigFileApplicationListener : Skipped config file 'file:./application-MyTest_LowerImportance.yml' (file:./application-MyTest_LowerImportance.yml) resource not found
TRACE 93941 --- [ main] o.s.b.c.c.ConfigFileApplicationListener : Skipped config file 'file:./application-MyTest_LowerImportance.yaml' (file:./application-MyTest_LowerImportance.yaml) resource not found
TRACE 93941 --- [ main] o.s.b.c.c.ConfigFileApplicationListener : Skipped config file 'classpath:/config/application-MyTest_MostImportant.properties' resource not found
TRACE 93941 --- [ main] o.s.b.c.c.ConfigFileApplicationListener : Skipped config file 'classpath:/config/application-MyTest_MostImportant.xml' resource not found
TRACE 93941 --- [ main] o.s.b.c.c.ConfigFileApplicationListener : Skipped config file 'classpath:/config/application-MyTest_MostImportant.yml' resource not found
TRACE 93941 --- [ main] o.s.b.c.c.ConfigFileApplicationListener : Skipped config file 'classpath:/config/application-MyTest_MostImportant.yaml' resource not found
TRACE 93941 --- [ main] o.s.b.c.c.ConfigFileApplicationListener : Skipped config file 'classpath:/config/application-MyTest_LowerImportance.properties' resource not found
TRACE 93941 --- [ main] o.s.b.c.c.ConfigFileApplicationListener : Skipped config file 'classpath:/config/application-MyTest_LowerImportance.xml' resource not found
TRACE 93941 --- [ main] o.s.b.c.c.ConfigFileApplicationListener : Skipped config file 'classpath:/config/application-MyTest_LowerImportance.yml' resource not found
TRACE 93941 --- [ main] o.s.b.c.c.ConfigFileApplicationListener : Skipped config file 'classpath:/config/application-MyTest_LowerImportance.yaml' resource not found
TRACE 93941 --- [ main] o.s.b.c.c.ConfigFileApplicationListener : Skipped config file 'classpath:/application-MyTest_MostImportant.properties' resource not found
TRACE 93941 --- [ main] o.s.b.c.c.ConfigFileApplicationListener : Skipped config file 'classpath:/application-MyTest_MostImportant.xml' resource not found
DEBUG 93941 --- [ main] o.s.b.c.c.ConfigFileApplicationListener : Loaded config file 'file:/Users/xxx/dev/myproject/target/test-classes/application-MyTest_MostImportant.yml' (classpath:/application-MyTest_MostImportant.yml)
TRACE 93941 --- [ main] o.s.b.c.c.ConfigFileApplicationListener : Skipped config file 'classpath:/application-MyTest_MostImportant.yaml' resource not found
TRACE 93941 --- [ main] o.s.b.c.c.ConfigFileApplicationListener : Skipped config file 'classpath:/application-MyTest_LowerImportance.properties' resource not found
TRACE 93941 --- [ main] o.s.b.c.c.ConfigFileApplicationListener : Skipped config file 'classpath:/application-MyTest_LowerImportance.xml' resource not found
DEBUG 93941 --- [ main] o.s.b.c.c.ConfigFileApplicationListener : Loaded config file 'file:/Users/xxx/dev/myproject/target/test-classes/application-MyTest_LowerImportance.yml' (classpath:/application-MyTest_LowerImportance.yml)
TRACE 93941 --- [ main] o.s.b.c.c.ConfigFileApplicationListener : Skipped config file 'classpath:/application-MyTest_LowerImportance.yaml' resource not found
TRACE 93941 --- [ main] o.s.b.c.c.ConfigFileApplicationListener : Skipped config file 'file:./config/application-MyTest_MostImportant-test.properties' (file:./config/application-MyTest_MostImportant-test.properties) resource not found

A modified version of @mxsb solution that allows us to define multiple files and in my case these are yml files.

In my application-dev.yml, I added this config that allows me to inject all the yml that have -dev.yml in them. This can be a list of specific files also.
"classpath:/test/test.yml,classpath:/test2/test.yml"

application:
properties:
locations: "classpath*:/**/*-dev.yml"
This helps to get a properties map.

@Configuration

public class PropertiesConfig {

private final static Logger LOG = LoggerFactory.getLogger(PropertiesConfig.class);

@Value("${application.properties.locations}")
private String[] locations;

@Autowired
private ResourceLoader rl;

@Bean
Map<String, Properties> myProperties() {
return stream(locations)
.collect(toMap(filename -> filename, this::loadProperties));
}

private Properties loadProperties(final String filename) {

YamlPropertySourceLoader loader = new YamlPropertySourceLoader();
try {
final Resource[] possiblePropertiesResources = ResourcePatternUtils.getResourcePatternResolver(rl).getResources(filename);
final Properties properties = new Properties();
stream(possiblePropertiesResources)
.filter(Resource::exists)
.map(resource1 -> {
try {
return loader.load(resource1.getFilename(), resource1);
} catch (IOException e) {
throw new RuntimeException(e);
}
}).flatMap(l -> l.stream())
.forEach(propertySource -> {
Map source = ((MapPropertySource) propertySource).getSource();
properties.putAll(source);
});

return properties;
} catch (IOException e) {
throw new RuntimeException(e);
}
}
}

However, if like in my case, I wanted to have to split yml files for each profile and load them and inject that directly into spring configuration before beans initialisation.

config
- application.yml
- application-dev.yml
- application-prod.yml
management
- management-dev.yml
- management-prod.yml
... you get the idea


The component is slightly different

@Component
public class PropertiesConfigurer extends PropertySourcesPlaceholderConfigurer
implements EnvironmentAware, InitializingBean {

private final static Logger LOG = LoggerFactory.getLogger(PropertiesConfigurer.class);

private String[] locations;

@Autowired
private ResourceLoader rl;
private Environment environment;

@Override
public void setEnvironment(Environment environment) {
// save off Environment for later use
this.environment = environment;
super.setEnvironment(environment);
}

@Override
public void afterPropertiesSet() throws Exception {
// Copy property sources to Environment
MutablePropertySources envPropSources = ((ConfigurableEnvironment) environment).getPropertySources();
envPropSources.forEach(propertySource -> {
if (propertySource.containsProperty("application.properties.locations")) {
locations = ((String) propertySource.getProperty("application.properties.locations")).split(",");
stream(locations).forEach(filename -> loadProperties(filename).forEach(source ->{
envPropSources.addFirst(source);
}));
}
});
}


private List<PropertySource> loadProperties(final String filename) {
YamlPropertySourceLoader loader = new YamlPropertySourceLoader();
try {
final Resource[] possiblePropertiesResources = ResourcePatternUtils.getResourcePatternResolver(rl).getResources(filename);
final Properties properties = new Properties();
return stream(possiblePropertiesResources)
.filter(Resource::exists)
.map(resource1 -> {
try {
return loader.load(resource1.getFilename(), resource1);
} catch (IOException e) {
throw new RuntimeException(e);
}
}).flatMap(l -> l.stream())
.collect(Collectors.toList());
} catch (IOException e) {
throw new RuntimeException(e);
}
}
}

I ran into a lot of issues when trying to figure this out. Here is my setup,

Dev Env : Windows 10,
Java : 1.8.0_25,
Spring Boot : 2.0.3.RELEASE,
Spring : 5.0.7.RELEASE

What I found is spring is sticking with the concept "Sensible defaults for configuration". What this translates in to is, you have to have all your property files as part of your war file. Once in there, you can then override them using the "--spring.config.additional-location" command line property to point to external property files. But this will NOT WORK if the property files are not part of the original war file.

Demo code:

[To see links please register here]