FILE on DecipherMiddleware

Demystifying file read as attachment in SOA 12c

Sat, 27 Nov 2021 21:12:00 +0000

In cases when we just want to move a file(without transformation) from one SFTP/FTP/file location to the other SFTP/FTP/file location, we can read a file with or without schema and then perform the write operation to the target location. If we provide an exact file schema for conversion to XML, it takes a lot of heap memory to transform from a csv or ffd to an XML document. In cases when files are not simple text files or schema is not available, one way to do so is to read the file as an opaque object and write it to a target location. In doing so, we still use a heap memory of the WebLogic server. If we are dealing with very large files, this may impact server performance as the conversion to opaque elements will take place behind the scenes. Using opaque schema is never recommended to process huge files. One better way of handling such a scenario is to read a file as an attachment in SOA 12c. Generally, we have below question:

What is reading a file as an attachment means? 🔗

In this case, we are not loading the contents of the file in the memory of the Weblogic server i.e in BPEL flow we will not see the contents of the file loaded instead, a href to the actual file is seen. In this case, the Weblogic server leverages SOA-INFRA tables to load file data in the table and in return gives href to the flow. This href actually links to data stored in the SOA-INFRA table and can be used to write files to the different locations (in the same or different FTP servers).

To understand better, let’s take an example of flow which polls the FTP directory every 10 minutes and read as an attachment. Then, how we can find the payload from SOA-INFRA tables.

Below are the steps to configure the file/FTP adapter in BPEL(SOA 12c).

Configuration Steps: 🔗

1. Create a SOA project in Jdevloper 12c.

2. Drag and drop FTP adapter from component pallet. Enter the service name. Click Next.

3. Select Define from operation and schema (specified later). Click Next.

4. Specify the source FTP JNDI name as configured in the Weblogic server. Click next.

5. Select operation type as Get File. Check the Read File As Attachment. If we know the character set, encoding, and Content-Type of the file, then specify else leave it blank. Click Next.

6. Specify the input directory from where we want to pick the file from. Check Archive processed files , and specify a directory for archive files if we want to archive the data after processing. Check Delete files after successful retrieval if we want to delete the file after processing. Click Next.

7. Provide the file name pattern or regex file name pattern and click Next.

8. Specify the polling frequency and click Next. Click on Finish to complete the adapter configuration.

9. Add a BPEL component with no service, i.e. define service later.

10. Join the wire from the FTP adapter to the BPEL component.

11. Open the BPEL component, and configure receive activity as shown below.

12. Click OK. Save and deploy the code on the Weblogic server.

Analyzing the read file as an attachment 🔗

Let’s test the service and see the behaviour of file being read as an attachment.

1. Place the file in the FTP location and wait for it to get it picked.

2. Once the file is picked, log in to the em console. We will see an instance created.

3. Open the flow, here we will see only the href.

Now after seeing this, we might think about where did our data go. As we see only href. But where is our actual data?

Our data is safe within soa-infra database.

4. To query the file data and file attributes, connect to the SOA-INFRA database of the SOA instance.

5. Run the below query

select * from attachment where dbkey='href-value';

File data is stored in blob format within the attachment table.

References: 🔗

https://docs.oracle.com/middleware/1213/adapters/develop-soa-adapters/GUID-088EFFE9-AE8B-4A35-B9DB-0A0DDA1BBBE7.htm#TKADP590

Please share your valuable feedback 😊😊😊

Reading .csv files with and without headers using different delimiters in Mule 4

Wed, 18 Aug 2021 18:29:00 +0000

Flat files especially .csv files are very common for data loading and data transfers. We generally come across scenarios where we want to read a .csv file, which may or may not contain the header row. Also, there is a chance .csv files don’t have ‘,’(comma) as delimiter rather has ‘’|"(pipe) as a delimiter or any other character as a delimiter. This article will discuss various ways to read such files in Mule 4 and create a JSON output of it.

Mule Flow 🔗

This will be a simple flow, where we will be reading a file using a file connector. Then convert the read data to JSON using transform message. Finally, log the JSON payload to the console using Logger. Here, we are not doing any complex transformation, it will be more like a MIME type conversion from application/csv to application/json, for better understanding the structure of input data.

Steps: 🔗

Create a flow in your mule application.
Drag and drop file connector with operation “On New or Updated file” on the source of mule flow.
In file connector properties under the general tab, specify or create a new connector configuration. Provide input directory, and polling frequency(here it is kept as 10 seconds). Specify the post-processing action like deleting the file, moving the file, or renaming the file. As per the below configurations, the mule application will look for a file in the inbound folder every 10 seconds. If the file is available, then it will process and move the file to the archive directory, deleting the file from the inbound location.

Drag and drop “Transform Message” from mule palette, and have output mime type as “application/json” as below.

Drag and drop logger from mule palette. Here we will be logging payload after transformation.
The final mule flow will look as below.

.csv file having header row in the file content 🔗

Let us take below employeeData.csv file as an example. For simplicity, we are taking dummy data of only 5 employees.

id,first name,last name,email,gender,designation
1,Dom,Lissett,dlissett0@test.com,Male,Recruiter
2,Susan,Moogan,smoogan1@test.com,Female,Executive Secretary
3,Carl,Cotesford,lcotesford2@test.com,Male,Tech Architect
4,Debor,Sprakes,dsprakes3@test.com,Female,Account Coordinator
5,Priscella,Genny,pgenny5@test.com,Female,Environmental Tech

In this example, data in the first row signifies the headers separated by “,” as a delimiter. Rest all other rows represent employee data.

Since the above .csv file contains a header row, all we need to do is to set the MIME type of file connector as “application/csv”. To do so, click on the file connector. Click on MIME Type and select “application/csv” from the drop-down list of values.

Place the file as defined in the file connector. Run the application. We will see the JSON response logged in the console as shown below.

INFO 2021-07-27 23:08:55,916 [[MuleRuntime].uber.03: [csv-read].csv-readFlow.CPU_INTENSIVE @1e42a374] [processor: csv-readFlow/processors/1; event: 895d1200-efb1-11eb-8134-d4d25289cb8d] org.mule.runtime.core.internal.processor.LoggerMessageProcessor: [
 {
 "id": "1",
 "first name": "Dom",
 "last name": "Lissett",
 "email": "dlissett0@test.com",
 "gender": "Male",
 "designation": "Recruiter"
 },
 {
 "id": "2",
 "first name": "Susan",
 "last name": "Moogan",
 "email": "smoogan1@test.com",
 "gender": "Female",
 "designation": "Executive Secretary"
 },
 {
 "id": "3",
 "first name": "Carl",
 "last name": "Cotesford",
 "email": "lcotesford2@test.com",
 "gender": "Male",
 "designation": "Tech Architect"
 },
 {
 "id": "4",
 "first name": "Debor",
 "last name": "Sprakes",
 "email": "dsprakes3@test.com",
 "gender": "Female",
 "designation": "Account Coordinator"
 },
 {
 "id": "5",
 "first name": "Priscella",
 "last name": "Genny",
 "email": "pgenny5@test.com",
 "gender": "Female",
 "designation": "Environmental Tech"
 }
]

Data in the first row became the keys for the JSON output.

.csv file without header row in the file content 🔗

Let us assume there is no header row in the content of employeeData.csv

1,Dom,Lissett,dlissett0@test.com,Male,Recruiter
2,Susan,Moogan,smoogan1@test.com,Female,Executive Secretary
3,Carl,Cotesford,lcotesford2@test.com,Male,Tech Architect
4,Debor,Sprakes,dsprakes3@test.com,Female,Account Coordinator
5,Priscella,Genny,pgenny5@test.com,Female,Environmental Tech

If we try to run with the same configurations, it will take the first row as keys and console output will look like.

INFO 2021-07-27 23:21:15,609 [[MuleRuntime].uber.03: [csv-read].csv-readFlow.CPU_INTENSIVE @1e42a374] [processor: csv-readFlow/processors/1; event: 4254cbd0-efb3-11eb-8134-d4d25289cb8d] org.mule.runtime.core.internal.processor.LoggerMessageProcessor: [
 {
 "1": "2",
 "Dom": "Susan",
 "Lissett": "Moogan",
 "dlissett0@test.com": "smoogan1@test.com",
 "Male": "Female",
 "Recruiter": "Executive Secretary"
 },
 {
 "1": "3",
 "Dom": "Carl",
 "Lissett": "Cotesford",
 "dlissett0@test.com": "lcotesford2@test.com",
 "Male": "Male",
 "Recruiter": "Tech Architect"
 },
 {
 "1": "4",
 "Dom": "Debor",
 "Lissett": "Sprakes",
 "dlissett0@test.com": "dsprakes3@test.com",
 "Male": "Female",
 "Recruiter": "Account Coordinator"
 },
 {
 "1": "5",
 "Dom": "Priscella",
 "Lissett": "Genny",
 "dlissett0@test.com": "pgenny5@test.com",
 "Male": "Female",
 "Recruiter": "Environmental Tech"
 }
]

By default, after setting MIME type to “application/json”, mule considers the first row as the header row. In such cases, we don’t want to consider the first row as the header row.

Click on the file connector. Click on MIME Type. Click on add parameter. Set key as “header” and value as “false”. Save the changes.

Now, by using the same file, we will be able to read data properly. But in this case, autogenerated values for keys like coloumn_0, column_1, etc. will be populated. Below is a sample log snippet.

INFO 2021-07-27 23:35:57,158 [[MuleRuntime].uber.05: [csv-read].csv-readFlow.CPU_INTENSIVE @6fbaa2fe] [processor: csv-readFlow/processors/1; event: 4fbf0590-efb5-11eb-8134-d4d25289cb8d] org.mule.runtime.core.internal.processor.LoggerMessageProcessor: [
 {
 "column_0": "1",
 "column_1": "Dom",
 "column_2": "Lissett",
 "column_3": "dlissett0@test.com",
 "column_4": "Male",
 "column_5": "Recruiter"
 },
 {
 "column_0": "2",
 "column_1": "Susan",
 "column_2": "Moogan",
 "column_3": "smoogan1@test.com",
 "column_4": "Female",
 "column_5": "Executive Secretary"
 },
 {
 "column_0": "3",
 "column_1": "Carl",
 "column_2": "Cotesford",
 "column_3": "lcotesford2@test.com",
 "column_4": "Male",
 "column_5": "Tech Architect"
 },
 {
 "column_0": "4",
 "column_1": "Debor",
 "column_2": "Sprakes",
 "column_3": "dsprakes3@test.com",
 "column_4": "Female",
 "column_5": "Account Coordinator"
 },
 {
 "column_0": "5",
 "column_1": "Priscella",
 "column_2": "Genny",
 "column_3": "pgenny5@test.com",
 "column_4": "Female",
 "column_5": "Environmental Tech"
 }
]

.csv file without header row and having a “|” as a delimiter 🔗

If we replace “,” delimiter with “|” delimiter in employeeData.csv

1|Dom|Lissett|dlissett0@test.com|Male|Recruiter
2|Susan|Moogan|smoogan1@test.com|Female|Executive Secretary
3|Carl|Cotesford|lcotesford2@test.com|Male|Tech Architect
4|Debor|Sprakes|dsprakes3@test.com|Female|Account Coordinator
5|Priscella|Genny|pgenny5@test.com|Female|Environmental Tech

On running the same mule application with the above data, the output of the above employeeData.csv file will be.

INFO 2021-07-27 23:42:06,938 [[MuleRuntime].uber.05: [csv-read].csv-readFlow.CPU_INTENSIVE @6fbaa2fe] [processor: csv-readFlow/processors/1; event: 2c2f0f20-efb6-11eb-8134-d4d25289cb8d] org.mule.runtime.core.internal.processor.LoggerMessageProcessor: [
 {
 "column_0": "1|Dom|Lissett|dlissett0@test.com|Male|Recruiter"
 },
 {
 "column_0": "2|Susan|Moogan|smoogan1@test.com|Female|Executive Secretary"
 },
 {
 "column_0": "3|Carl|Cotesford|lcotesford2@test.com|Male|Tech Architect"
 },
 {
 "column_0": "4|Debor|Sprakes|dsprakes3@test.com|Female|Account Coordinator"
 },
 {
 "column_0": "5|Priscella|Genny|pgenny5@test.com|Female|Environmental Tech"
 }
]

Now, we are getting only one key-value pair for each row. By default mule considers “,” as delimiter for “application/csv” MIME type data. We need to explicitly define “|” as a delimiter in parameters under the MIME type.

Click on the file connector. Click on MIME Type. Click on add parameter. Set Key as “separator” and Value as “|”. Save the changes.

Now, by using the same file, we will be able to read data properly as shown below.

INFO 2021-07-27 23:51:57,557 [[MuleRuntime].uber.06: [csv-read].csv-readFlow.CPU_INTENSIVE @6efd4175] [processor: csv-readFlow/processors/1; event: 8c31a710-efb7-11eb-8134-d4d25289cb8d] org.mule.runtime.core.internal.processor.LoggerMessageProcessor: [
 {
 "column_0": "1",
 "column_1": "Dom",
 "column_2": "Lissett",
 "column_3": "dlissett0@test.com",
 "column_4": "Male",
 "column_5": "Recruiter"
 },
 {
 "column_0": "2",
 "column_1": "Susan",
 "column_2": "Moogan",
 "column_3": "smoogan1@test.com",
 "column_4": "Female",
 "column_5": "Executive Secretary"
 },
 {
 "column_0": "3",
 "column_1": "Carl",
 "column_2": "Cotesford",
 "column_3": "lcotesford2@test.com",
 "column_4": "Male",
 "column_5": "Tech Architect"
 },
 {
 "column_0": "4",
 "column_1": "Debor",
 "column_2": "Sprakes",
 "column_3": "dsprakes3@test.com",
 "column_4": "Female",
 "column_5": "Account Coordinator"
 },
 {
 "column_0": "5",
 "column_1": "Priscella",
 "column_2": "Genny",
 "column_3": "pgenny5@test.com",
 "column_4": "Female",
 "column_5": "Environmental Tech"
 }
]

References: 🔗

https://docs.mulesoft.com/mule-runtime/4.3/dataweave-formats-csv

Please share your valuable feedback 😊

This is a repost of the LinkedIn article….