IPF 6.0 Migration Guide
IPF 6.0 comes with some significant changes that must be considered when upgrading from IPF 5.3.
Jakarta EE 11 and Spring Boot 4.1
IPF 6.0 moves to the Jakarta EE 11 stack. Java 17 remains the minimum (the build is also tested against 21 and 25), but the surrounding libraries move on:
| IPF 5.3.0 | IPF 6.0.0 | |
|---|---|---|
| Spring Boot | 3.5 | 4.1 |
| Apache Camel | 4.18 | 4.22 |
| Apache CXF | 4.1 | 4.2 |
| Jackson | 2.22 | 3.1 |
| Kotlin | 2.2 | 2.4 |
| HAPI FHIR | 8.10 | 8.12 |
| Groovy | 5.0 | 5.1 |
HAPI HL7v2 (2.6) is unchanged. Consequences for applications:
- Jackson 3. The databind API moved from
com.fasterxml.jackson.*totools.jackson.*. Do not raisejackson-versionpast the version Spring Boot manages — Jackson 3.2.x rejects the type-id/bean-property name collision that the XDS metadata model relies on, and the build will fail.httpclient5-versionandtomcat-versionare likewise kept in sync with Spring Boot’s managed versions, whilekotlin-versiondeliberately overrides Spring Boot’s so that the Kotlin compiler and the stdlib stay on the same release. Each of them carries an explanatory comment in the pom. -
Generic type parameters were removed from a large number of public types, because they carried no information that callers could use. Applications that implement or extend these SPIs have to drop the type arguments — a source-only change, no behavior is affected. Among the types concerned are
AuditStrategy,AuditStrategySupport,AuditInterceptor,DispatchAuditStrategy,NoAuditStrategy,NakFactory,AsynchronyCorrelator,ClientRequestFactory,FhirAuditStrategy,AbstractFhirAuditStrategy,FhirEndpoint,FhirEndpointConfiguration,FhirInteractionId,FhirTransactionConfiguration,FhirTransactionOptionsProvider,FhirComponentWithOptions,HL7v2Endpoint,Hl7v2InteractionId,Hl7v2TransactionConfiguration,Hl7v2ConfigurationHolder,Hl7v3Component,Hl7v3Endpoint,Hl7v3InteractionId,MllpComponent,MllpEndpoint,AbstractWsEndpointandHpdEndpoint:// before public class MyRequestFactory implements ClientRequestFactory<IClientExecutable<IOperationUntypedWithInput<Bundle>, ?>> { public IClientExecutable<IOperationUntypedWithInput<Bundle>, ?> getClientExecutable(...) // after public class MyRequestFactory implements ClientRequestFactory { public IOperationUntypedWithInput<Bundle> getClientExecutable(...)ClientRequestFactory#getClientExecutablenow declares the concrete HAPI client type instead of a wildcardIClientExecutable, so implementations can return their builder directly. - ATNA audit records changed where the latest DICOM CPs required it. See ATNA auditing below.
ATNA auditing
The audit record model and builders follow the latest DICOM CPs. Tests that assert on the exact content of generated audit messages need to be re-baselined.
- Deprecated transports removed. The
AuditTransmissionChannelconstantsNIO-UDP,VERTX-UDP,NIO-TLSandFHIR-REST-TLS, deprecated since IPF 5.0, are gone. A configuration such asauditRepositoryTransport=NIO-TLS(oripf.atna.audit-repository-transport=NIO-TLSin Spring Boot) no longer starts. UseNETTY-TLSorREACTOR-NETTY-TLSinstead ofNIO-TLS,UDPinstead ofNIO-UDPandVERTX-UDP, andFHIR-REST-APACHE5-TLSorFHIR-REST-METHANOL-TLSinstead ofFHIR-REST-TLS. See Audit Transmission Protocols for all transports and the IPF versions supporting them. DicomInstancesTransferredBuilder:setSendingProcessParticipantandsetReceivingProcessParticipantare merged intosetProcessParticipant.- New codes.
ActiveParticipantRoleIdCodegainsInitiator(DCM 110156) andPerson(SNOMED CT 125676002);MediaTypegainsClipboardManagerandMessagingSystem.MediaType.EmailMessageis deprecated. - Defaults for names. If no participant object name is given, the participant object ID is used instead; if no patient name is given, the patient ID is used.
DataExportBuilderandDataImportBuilderhave new overloads ofsetDestinationMediaParticipant/setSourceMediaParticipantthat take theActiveParticipantRoleIdCodeexplicitly.
Removal of FHIR STU3 support
Support for FHIR STU3 (DSTU3) has been removed. The modules ipf-commons-ihe-fhir-stu3-*,
ipf-platform-camel-ihe-fhir-stu3-* and ipf-fhir-stu3-spring-boot-starter no longer exist; use
their FHIR R4 counterparts instead. The default FhirContext of the Camel FHIR data formats is
now R4 instead of DSTU3.
MHD 4.2.4
MHD support was updated to version 4.2.4, remaining backwards compatible with 4.2.3.
MhdVersionis now an enum. It used to be a marker interface with asupportsVersion()method and a nestedenum Version { v320, v423 }; it is now simplyenum MhdVersion { v320, v423, v424 }. The marker interfacesMhd423,Pdqm320andPixm310were removed along with it — a model class does not have a single version, since the resources it builds conform to more than one MHD version at once.- Profile validation against MHD 3.2.0 is no longer possible. The v320 implementation guide
resources are no longer shipped, and
MhdValidator.packagePathFor(MhdVersion.v320)raises anIllegalArgumentException. Thev320constant remains for the compatibility resource providers, andIti66Options.COMPATIBILITYstill serves bothDocumentManifestandListresources. -
entryUUID and uniqueId are distinguished by
Identifier.typerather thanIdentifier.use(CP-ITI-1328-01), expressed by the newMhdIdentifierTypeenum. For compatibility with MHD 4.2.3 the model classes keep setting the oldIdentifier.use—officialfor the entryUUID,usualfor the uniqueId — so code reading theusecontinues to work. New code should match on the type instead:MhdIdentifierType.ENTRY_UUID.find(documentReference.getIdentifier()) .map(Identifier::getValue) .ifPresent(entryUuid -> ...); - The Target Communities Option (CP-ITI-1326-02) is supported. The
DocumentReferenceandListmodel classes carry ahomeCommunityIdextension, and ITI-66 and ITI-67 accept atargetCommunityIdListsearch parameter. - Canonical references that pin a version, e.g.
.../Encounter|4.0.1, are resolved again during profile validation.
Hardened XML processing
XML processing was further hardened against XXE and SSRF, which restricts what the parsers will fetch.
- The new
org.openehealth.ipf.commons.xml.XmlSecurityapplies the countermeasures to the JAXP and Saxon components that see untrusted input:XsdValidator,XsltTransmogrifier,SchematronTransmogrifier,XqjTransmogrifierandCDAR2Parser. External entities and DTDs are no longer resolved. Schema compilation is restricted to thefileandjarprotocols, so anxs:importorxs:includemay still reach a sibling file or a resource inside a jar but not the network; instance validation and XSLT, Schematron and XQuery processing permit no external access at all. Anything that relied on resolving external entities, on a remotexs:import, or ondocument()over http has to provide those resources locally. - The deprecated classes
org.openehealth.ipf.commons.core.ContentMapandorg.openehealth.ipf.commons.core.DomBuildersPoolwere removed. The pool lives on asorg.openehealth.ipf.commons.xml.DomBuildersPoolinipf-commons-xml.ipf-commons-coreno longer depends onjaxb-runtime, not even optionally. - The object pools of
DomBuildersPoolandJaxWsClientFactorynow evict idle entries, using the neworg.openehealth.ipf.commons.core.pool.PoolEvictor. No configuration is required. XmlYieldermoved fromipf-commons-xmltoipf-commons-ihe-hl7v3, packageorg.openehealth.ipf.commons.ihe.hl7v3, its only user.ipf-commons-xmlno longer requires Groovy at all, so modules that relied on its transitivegroovy-xmldependency must declare that themselves.
Tracing with Camel telemetry
IPF’s own tracing support has been replaced by Apache Camel’s telemetry SPI (camel-telemetry).
Removed: org.openehealth.ipf.commons.ihe.hl7v2.tracing.MessageTracer together with
Hl7MessageSetter and Hl7MessageGetter, the interceptors ConsumerTracingInterceptor and
ProducerTracingInterceptor, the B3 header constants SPAN_ID, TRACE_ID, PARENT_SPAN_ID,
SAMPLED and FLAGS in org.openehealth.ipf.commons.ihe.core.Constants, and IPF’s dependency on
micrometer-tracing.
New: every eHealth transaction component ships a Camel SpanDecorator, registered through
META-INF/services/org.apache.camel.telemetry.SpanDecorator. When a camel-telemetry based
tracer is active in the Camel context, the spans Camel creates for IPF endpoints carry the
metadata of the transaction. Nothing has to be configured on the endpoints.
Trace context is propagated differently per transport:
-
HTTP and SOAP transactions leave propagation to the underlying libraries; the Camel span joins the context they establish. For the SOAP-based transactions, each transaction family starter (
ipf.xds,ipf.hl7v3,ipf.hpd,ipf.svs,ipf.xacml20,ipf.ech) has a newobserving...property. If set, it exposes the CXF beansobservationFeatureandobservationClientFeatureover Spring Boot’sObservationRegistry; this requirescxf-integration-tracing-micrometeron the classpath. Attach them per endpoint:from("xds-iti18:my-registry?features=#observationFeature") .to("xds-iti18://peer:8080/registry?features=#observationClientFeature"); -
HL7v2 over MLLP has no transport-level mechanism, so the trace context still travels in a
ZTRsegment, one repetition of field 1 per propagation entry (ZTR|traceparent^00-0af7…-b7ad…-01). The newProducerTraceContextInterceptorwrites it and is part of the default producer chain, so no interceptor has to be configured any more. It does nothing unless telemetry is active.
To migrate, delete the MessageTracer, ConsumerTracingInterceptor.Factory and
ProducerTracingInterceptor.Factory beans and their interceptorFactories references on the
endpoint URIs, and activate a camel-telemetry based tracer instead. We recommend
camel-opentelemetry2, which reports to OpenTelemetry and propagates W3C traceparent by default:
<dependency>
<groupId>org.apache.camel</groupId>
<artifactId>camel-opentelemetry2</artifactId>
<version>${camel-version}</version>
</dependency>
In Spring Boot, use org.apache.camel.springboot:camel-opentelemetry2-starter instead, which
activates the tracer automatically.
The old tracer used B3 multi-headers. The new one propagates whatever the configured tracer uses,
usually W3C traceparent, so traces are not joined between an IPF 5.x and an IPF 6.0 peer over
MLLP.
Removal of the Configurer SPI
The Configurer SPI only ever expressed “collect all beans of type T and push them into some
holder”, which core Spring expresses as beanFactory.getBeanProvider(T.class).orderedStream().
The holders and registrars now collect their own contributions, so the SPI and the configurers
built on it are deprecated and will be removed in the next major release:
org.openehealth.ipf.commons.core.config.Configurerorg.openehealth.ipf.commons.core.config.OrderedConfigurerorg.openehealth.ipf.commons.spring.core.config.SpringConfigurationPostProcessororg.openehealth.ipf.commons.spring.map.config.CustomMappingsConfigurerorg.openehealth.ipf.modules.hl7.config.CustomModelClassFactoryConfigurer(Java and Kotlin)org.openehealth.ipf.commons.core.extend.config.DynamicExtensionConfigurer
Everything listed above keeps working in IPF 6.0, and the old and the new mechanism may be active at the same time without any contribution being applied twice. Spring Boot applications need no changes at all, apart from the exceptions described at the end of this section.
One thing does change for a context without Spring Boot: the Groovy and Kotlin mapping DSLs
('M'.map('gender') and the like) now resolve a bean of type Mappings rather than a
MappingService. A context declaring only a SpringBidiMappingService, like the “before” below,
starts, but every DSL call fails with NoSuchBeanDefinitionException. Declare a SpringMappings
bean, and construct the SpringBidiMappingService over it if code still needs the untyped
service — which is what the “after” below does.
The contribution beans themselves — CustomMappings, CustomModelClasses, DynamicExtension
and CustomRouteBuilder — are untouched.
Delete the post processor and drop the configurers
<!-- before -->
<bean id="mappingService" class="...spring.map.SpringBidiMappingService"/>
<bean id="c1" class="...spring.map.config.CustomMappingsConfigurer">
<property name="mappingService" ref="mappingService"/>
</bean>
<bean id="c2" class="...modules.hl7.config.CustomModelClassFactoryConfigurer">
<property name="customModelClassFactory" ref="customModelClassFactory"/>
</bean>
<bean id="c3" class="...core.extend.config.DynamicExtensionConfigurer"/>
<bean id="postProcessor" class="...spring.core.config.SpringConfigurationPostProcessor"/>
<!-- after -->
<bean id="mappings" class="...spring.map.SpringMappings"/>
<!-- only where code still needs the untyped MappingService -->
<bean id="mappingService" class="...spring.map.SpringBidiMappingService">
<constructor-arg ref="mappings"/>
</bean>
<bean class="...modules.hl7.config.CustomModelClassesRegistrar">
<property name="customModelClassFactory" ref="customModelClassFactory"/>
</bean>
<bean class="...spring.core.extend.SpringDynamicExtensionRegistrar"/>
| Old | New |
|---|---|
CustomMappingsConfigurer |
nothing — SpringMappings (and a SpringBidiMappingService that owns its mappings) collects the MappingResourceHolder beans itself |
CustomModelClassFactoryConfigurer |
org.openehealth.ipf.modules.hl7.config.CustomModelClassesRegistrar |
Kotlin CustomModelClassFactoryConfigurer |
org.openehealth.ipf.modules.hl7.kotlin.config.CustomModelClassesRegistrar |
DynamicExtensionConfigurer |
org.openehealth.ipf.commons.spring.core.extend.SpringDynamicExtensionRegistrar |
CustomRouteBuilderConfigurer |
the same class — keep the bean, it collects for itself now |
SpringConfigurationPostProcessor |
delete it, but read the next section first |
A CustomModelClassesRegistrar without a customModelClassFactory property configures every
CustomModelClassFactory bean of the application context.
Declare <ipf:globalContext/> if you do not already have it
SpringConfigurationPostProcessor created a SpringRegistry as a side effect, and thereby
initialized the ContextFacade that the stateful Groovy and Kotlin extensions look beans up in.
Dropping the post processor without declaring the registry explicitly results in
IllegalStateException: Registry instance has not been set at the first extension call:
<beans xmlns:ipf="http://openehealth.org/schema/ipf-commons-core"
xsi:schemaLocation="...
http://openehealth.org/schema/ipf-commons-core
http://openehealth.org/schema/ipf-commons-core.xsd">
<ipf:globalContext/>
Ordering moves onto the contribution
setOrder(...) on a configurer applied a single order value to everything that configurer
collected. Use Spring’s @Order / Ordered on the contributing bean instead, which is more
fine grained. @Order on a @Bean factory method is honoured, as is @Order on the
contribution class or an Ordered implementation:
@Bean
@Order(10)
CustomMappings baseMappings() { ... }
Contributions of ancestor application contexts are collected as well. Without an explicit
@Order the contributions of the local context come first; use @Order if that sequence
matters.
Behavioral changes
- Contributions are collected earlier. Collecting now happens once all singletons have been
instantiated (
SmartInitializingSingleton), and for route builders when Spring starts itsLifecyclebeans — no longer onContextRefreshedEvent. Mapping definitions and custom HL7v2 model classes are therefore in place before the Camel routes are built and started. They are still not in place during bean initialization: a@PostConstructmethod orafterPropertiesSet()sees only the resources set on the mappings bean directly, not those contributed throughCustomMappings. - Groovy DSL extensions are still registered before the route builders build their routes. This is now guaranteed by the Spring lifecycle rather than by bean registration order.
CustomRouteBuilderConfigurerlooks up theCamelContextitself if the application context contains exactly one, so itscamelContextproperty has become optional.
Exceptions for Spring Boot applications
- If the application declares its own
CustomModelClassFactorybean, it must be constructed with a mutable map, e.g.new HashMap<>(Map.of(...))rather thanMap.of(...). HAPI adds the contributed package definitions straight into that map, so an immutable one lets the context fail to start with anUnsupportedOperationException. - The
CustomMappingsConfigurerandSpringConfigurationPostProcessorbeans ofIpfAutoConfigurationare gone. Inject theSpringMappingsbean instead — or theSpringBidiMappingServiceover it, which is deprecated but still auto-configured. - The auto-configured
SpringMappingsbacks off as soon as the application declares any bean of typeMappings, since the mapping DSLs resolve that type and would otherwise find two. An application declaring its ownSpringBidiMappingServiceshould construct it over theSpringMappingsbean. - IPF’s own
CustomMappingscontributions are read first, so an application’sCustomMappingscan override a shipped mapping or delegate to it, whatever order it declares.
Reimplemented mapping service
The mapping service has been reimplemented around a typed, Optional-returning Mappings API
and declarative mapping formats (IPF XML, IPF YAML, FHIR ConceptMap R4). ipf-commons-map is now
a parent POM; depend on ipf-commons-map-core plus one module per format you read.
MappingService, BidiMappingService, SpringBidiMappingService and the Groovy .map DSL are
deprecated and may be removed in IPF 7.0, but keep working in 6.x. See
Defining and Loading Mappings for the
formats, the Spring wiring and MappingConverter, which converts .map scripts.
Composite values are gone
The ~ convention of IPF 5.x no longer exists, neither in the Groovy and Kotlin mapping DSLs nor
in the deprecated BidiMappingService: a Collection is no longer joined into a key, and a value
containing ~ is returned as one string rather than being split into a List. map/mapReverse on a
Collection are gone, and BidiMappingService rejects a Collection key. Split such a mapping
into one mapping per component, or join and split the parts in the calling code:
// IPF 5.x: a List
[app, facility].map('device_OID')
// IPF 6.0
"${app}~${facility}".map('device_OID')?.split('~')
The Groovy .map loader logs a warning for each mapping that uses the convention.
For the same reason, the HL7 v2/v3 translators ask two new mappings instead of
hl7v2v3-interactionId-eventStructure, whose values were composites like A01~ADT_A01:
hl7v2v3-interactionId-triggerEvent (A01) and hl7v2v3-interactionId-messageStructure
(ADT_A01). map('…eventStructure')[0] becomes map('…triggerEvent'). Both mappings are in
hl7-v2-v3-translation.mapping.xml, and in the legacy hl7-v2-v3-translation.map as well, so a
configuration still reading the latter keeps working. A copy of that file an application
maintains itself needs the two mappings added.
Empty values are literal
An entry mapping to '' now always returns '' and never falls back to the mapping’s ELSE
clause. A default argument applies only where there is no value at all. In IPF 5.x such an entry
was never returned: Groovy’s ?: treated '' as no value, so the mapping’s ELSE applied, or
null without one. An empty result gave way to the default argument of map(mapping, default)
or get(mapping, key, default). With 'N' : '' and (ELSE) : 'IMP', 'N'.map(…) used to give
'IMP' and now gives ''.
This holds for Mappings, the Groovy and Kotlin DSLs and the deprecated BidiMappingService
alike. Check mappings with empty values whose callers relied on the old result. Where an entry was
meant to fall back to (ELSE), remove it.
IPF’s own HL7 v2 → v3 translation shows the change: hl7v2v3-bidi-administrativeGender-administrativeGender
maps '' to '', so an empty PID-8 used to fall through to the reverse fallback and produce
<administrativeGenderCode code="UN"/>, and now produces no administrativeGenderCode element —
which is what the entry says. Remove the entry from a copy of the mapping to get UN back.
Typed mappings must be rewritten
A .map file is a script, and the old service held whatever objects it evaluated to — enum
constants, numbers, lists — and handed exactly those objects back. The new model holds strings
only, and so does every mapping format. A .map script is therefore still loaded, but each key,
value and fallback result is reduced to its toString() form, and the deprecated MappingService
returns that string rather than the object the script declared:
// default-enumerations.map
documentStatus(
(Enumerations.DocumentReferenceStatus.CURRENT) : AvailabilityStatus.APPROVED,
...
)
// IPF 5.x: the AvailabilityStatus constant. IPF 6.0: the String "APPROVED"
(AvailabilityStatus) mappingService.get("documentStatus", status) // ClassCastException
This compiles, and only fails when the lookup runs. toString() is not necessarily the constant
name either (ChronoUnit.SECONDS becomes "Seconds"), so converting back with Enum.valueOf is
not a general fix.
Such typed mappings cannot be expressed in any of the new formats and must be rewritten in Java —
for a table between two enums a switch expression is the natural replacement. Without a
default branch it fails to compile when either enum grows a constant. Mappings between plain
codes are unaffected.
To find them, look for warnings. The .map loader logs one at startup for every mapping whose
keys, values or constant fallbacks are not strings. For a fallback closure the first time it
computes a non-string result. MappingConverter reports the same mappings among its conversion
warnings.
HL7v2 DSL (Groovy and Kotlin)
The Groovy and Kotlin HL7v2 DSLs have been aligned and a number of bugs fixed. Most code continues to work unchanged; check the following if you rely on the affected behavior.
Both DSLs
- Repetitions are no longer added accidentally. Holding on to a repeating element and accessing
the same repetition twice (
def f = pid[3]; f(0)[1] = 'a'; f(0)[1] = 'b') used to append a new repetition on each access. Now it addresses the same repetition. Accessing a repetition beyond the last one (e.g.NK1(3)when only one exists) now creates all missing repetitions in between, instead of silently returning the next free one. - Extra components are indexed starting with 1, like all other DSL indexes, and
.valuenow returns the first extra component (it previously returned the second).
Groovy DSL
- Repetition access
(n)now behaves like in the Kotlin DSL. On non-repeatable fields, segments and groups (e.g.msg.PID(0),pid[7](0)),(0)returns the element itself instead of throwing. The same applies to a repetition that is not the first one (e.g.msg.NK1(2)(0)), which no longer jumps to a sibling repetition; to select one, use the parent:msg.NK1(n). In both cases,(n)withn > 0throws anHL7DslException. - Responses created with
respond()/MessageUtils.response()now copy the complete sending/receiving application and facility fields (MSH-3..6, e.g. including universal ID and ID type) and the complete MSH-11 processing ID from the request. In particular, the processing mode is no longer forced toT. - Misspelled methods on HL7 types now raise a
MissingMethodException. Previously they were treated as mapping calls and failed with “Registry instance has not been set”.
Kotlin DSL
-
loadHl7()andmakeHl7()return a nullableT?(nullfor missing or empty content). Add!!or handlenullat the call site:val msg: Message = loadHl7(context, "msg.hl7")!!Resources are now resolved from the classloader root; a leading
/is ignored. - Accessing a component > 1 of a primitive (e.g.
msh[10][2]) returns aNullobject (empty,value == null) instead of throwing. This keeps expressions valid for HL7 versions in which a primitive field has become a composite. Code that caughtHl7DslExceptionfor this purpose should checkvalue == nullinstead. - Segments can be assigned from any
Structure, somsg1["EVN"] = msg2["EVN"]no longer needs a cast.
New features
FHIR Binary streaming uploads
Large document uploads with ITI-65 and ITI-105 no longer need several times the document size
on the heap: the document content can be split out of the request body into memory or a temporary
file before it reaches the FHIR parser. The feature is off by default and is enabled with
ipf.fhir.streaming-upload.enabled. See FHIR Binary streaming uploads for how it works, how to get
hold of the document content and all configuration properties.
PIXm Patient Identity Feed ITI-104
The new pixm-iti104 component implements the Patient Identity Feed FHIR (ITI-104) for both the Patient
Identity Source and the Patient Identifier Cross-reference Manager, including the Remove Patient Option,
validation against the PIXm and PDQm Patient profiles, and auditing according to the PIXm AuditEvent profiles.
Translators proxy ITI-104 to the PIX Feed (ITI-8), see Translation between FHIR and HL7 v2.
Other fixes
These fixes change behavior that an application may have relied on:
SslAwareApacheRestfulClient5Factoryhonours a configuredHostnameVerifier. Previously a verifier such asNoopHostnameVerifierwas ignored: httpclient5 defaulted toHostnameVerificationPolicy.BOTH, and JSSE rejected the hostname during the TLS handshake before the verifier was consulted.- FHIR paging. Further pages of a
LazyBundleProviderare loaded with a copy of the originalRequestDetailsinstead of seeing the servlet request and response of the first request, which are no longer valid by then. ipf.xds.cachingnow also picks up theCacheManagercontributed by Spring Boot’s cache auto-configuration, not only one declared by the application. Caching may therefore become active where it silently did nothing before.- The ATNA starter (and thus every transaction family starter) no longer fails to start when
spring-boot-actuatoris absent from the classpath. - The
mhd-iti68-binandfhirendpoint schemes resolve again. Their Camel component descriptors named the component classes in the wrong package, so routes using them failed to start.