Creating an OIA ETL JPP scheduler

After implementing OIA on a number of real-world engagements, I’ve slowly grown tired of the 3rd party clover ETL scheduler configured OOTB for data imports. From my experience, many organizations have a centralized ETL framework for manipulations so there’s no logical justification to segregate OIA’s required ETL efforts. But what if they’re missing an ETL framework? Were talking about the minority of companies here though here’s an alterative to scheduling imports into OIA.
March 22, 2013
OIA-11g

As clover ETL is a 3rd party technology, you are required to purchase at an independent license, probably the GUI desktop version for 1 system, costing a whopping $4000.

As an alterative, I have stewed more toward the concept of java pre-processors (JPPs) to accommodate the code modification and conformance checking. I find Individuals also prefer the scalability of java as a technology to clover.

https://technicalconfessions.com/images/postimages/postimages/_30_3_JPP_concept.png

The image presents the concept of a JPPs becoming the intermediate from converting any data feeds into the acceptable OIA format, which can then be dropped into the OIA staging repository in preparation to importing them into OIA.

https://technicalconfessions.com/images/postimages/postimages/_30_4_basic_ETL_scheduler.png

With this in mind, I have been suggesting this alternative to companies though I only recently came across the idea of java classes as java beans and allowing OIA’s spring framework to schedule the alteration.

https://technicalconfessions.com/images/postimages/postimages/_30_5_web_xml_alteration_in_OIA.png

For scheduling purposes, the java file should extend the TimerTask package which will make the manipulations required to then populate this into the OIA staging as a new file.

https://technicalconfessions.com/images/postimages/postimages/_30_6_explaination_of_bean.png

I have created a new file called jpp-etl.xml, which resides within the WEB-INF directory and altered the web.xml (to reference the jpp-etl.xml file). Once configured, OIA can then take the responsibility on the ETL effort without the dependency of OOTB clover graphs.

https://technicalconfessions.com/images/postimages/postimages/_30_7_adding_line_in_webxml.png

About the author

Daniel is a Technical Manager with over 10 years of consulting expertise in the Identity and Access Management space.
Daniel has built from scratch this blog as well as technicalconfessions.com
Follow Daniel on twitter @nervouswiggles

Comments

Other Posts

AWS-PHP integration - Email not sent. SMTP Error: Could not authenticate.

phpsmtpaws

February 6, 2020
Created by: Daniel Redfern
AS I was migrating my environment into an S3 environment, I wanted to leverage off the SES services that AWS provide, more specifically, to leverage the off the SMTP functionality by sending an email via PHP
Read More...

SOLUTION: no headers files (.h) found in softwareserial - Arduino

Arduino

February 24, 2019
Created by: Daniel Redfern
The WeMos D1 is a ESP8266 WiFi based board is an extension to the current out-of-the-box library that comes with the Arduino installation. Because of this, you need to import in the libraries as well as acknowledging the specific board. This process is highly confusion with a number of different individuals talking about a number of different ways to integrate.
Read More...

NameID element must be present as part of the Subject in the Response message

ShibbolethSAML

August 7, 2018
Created by: Daniel Redfern
NameID element must be present as part of the Subject in the Response message, please enable it in the IDP configuration.
Read More...

HOW TO provision AD group membership from OpenIDM

OpenIDMICFAD-connector

June 15, 2018
Created by: Daniel Redfern
For what I see, there's not too many supportive documentations out there that will demonstrate how provision AD group membership with the ICF connector using OpenIDM. The use of the special ldapGroups attribute is not explained anywhere in the Integrators guides to to the date of this blog. This quick blog identifies the tasks required to provision AD group membership from OpenIDM to AD using the LDAP ICF connector. However this doesn't really explain what ldapGroups actually does and there's no real worked example of how to go from an Assignment to ldapGroups to an assigned group in AD. I wrote up a wiki article for my own reference: AD group memberships automatically to users This is just my view, others may disagree, but I think the implementation experience could be improved with some more documentation and a more detailed example here.
Read More...

ForgeRock OpenIDM - InvalidCredentialException: Remote framework key is invalid

ICFIDMOpenIDMOpenICF

November 8, 2017
Created by: Daniel Redfern
In the past, the similar error occurred though for the Oracle Identity Management solution. invalidcredentialexception remote framework key is invalid Because they all share the ICF connector framework, the error/solution would be the same.
Read More...

org.forgerock.script.exception.ScriptCompilationException: missing ; before statement

IDMsync.confforgerockopenidm

November 8, 2017
Created by: Daniel Redfern
org.forgerock.script.exception.ScriptCompilationException: missing ; before statement
Read More...

ForgeRock IDM - org.forgerock.script.exception.ScriptCompilationException: missing ; before statemen

OpenIDMsync.confForgeRock

September 17, 2017
Created by: Daniel Redfern
ForgeRock IDM - org.forgerock.script.exception.ScriptCompilationException: missing ; before statement
Read More...

Caused by: org.forgerock.json.resource.BadRequestException: Target does not support attribute groups

OpenIDMForgeRockICFConnector

September 17, 2017
Created by: Daniel Redfern
When performing the attempt of a reconciliation from ForgeRock IDM to Active Directory, I would get the following error
Read More...

ForgeRock OpenIDM - InvalidCredentialException: Remote framework key is invalid

OpenIDMForgeRockICFConnectorAD

September 17, 2017
Created by: Daniel Redfern
In the past, the similar error occurred though for the Oracle Identity Management solution. invalidcredentialexception remote framework key is invalid Because they all share the ICF connector framework, the error/solution would be the same.
Read More...

ERROR Caused by com.google.api.client.auth.oauth2.TokenResponseException 400 Bad Request - invalid_g

OpenIDMIDMGoogleGoogle-AppsICFreconciliation

September 12, 2017
Created by: Daniel Redfern
During the reconcilation from OpenIDM to the ICF google apps connector, the following error response would occur. ERROR Caused by com.google.api.client.auth.oauth2.TokenResponseException 400 Bad Request - invalid_grant
Read More...