Demystifying file read as attachment in SOA 12c

 In cases when we just want to move a file(without transformation) from one SFTP/FTP/file location to the other SFTP/FTP/file location, we can read a file with or without schema and then perform the write operation to the target location. If we provide an exact file schema for conversion to XML, it takes a lot of heap memory to transform from a csv or ffd to an XML document. In cases when files are not simple text files or schema is not available, one way to do so is to read the file as an opaque object and write it to a target location. In doing so, we still use a heap memory of the WebLogic server. If we are dealing with very large files, this may impact server performance as the conversion to opaque elements will take place behind the scenes. Using opaque schema is never recommended to process huge files. One better way of handling such a scenario is to read a file as an attachment in SOA 12c. Generally, we have below question:

What is reading a file as an attachment means?

In this case, we are not loading the contents of the file in the memory of the Weblogic server i.e in BPEL flow we will not see the contents of the file loaded instead, a href to the actual file is seen. In this case, the Weblogic server leverages SOA-INFRA tables to load file data in the table and in return gives href to the flow. This href actually links to data stored in the SOA-INFRA table and can be used to write files to the different locations (in the same or different FTP servers). 

To understand better, let's take an example of flow which polls the FTP directory every 10 minutes and read as an attachment. Then, how we can find the payload from SOA-INFRA tables. 


Below are the steps to configure the file/FTP adapter in BPEL(SOA 12c).

Configuration Steps:

1. Create a SOA project in Jdevloper 12c.
2. Drag and drop FTP adapter from component pallet. Enter the service name. Click Next.
3. Select Define from operation and schema (specified later). Click Next.
4. Specify the source FTP JNDI name as configured in the Weblogic server. Click next.
5. Select operation type as Get File. Check the Read File As Attachment. If we know the character set, encoding, and Content-Type of the file, then specify else leave it blank. Click Next.
6. Specify the input directory from where we want to pick the file from. Check Archive processed files, and specify a directory for archive files if we want to archive the data after processing. Check Delete files after successful retrieval if we want to delete the file after processing. Click Next.
7. Provide the file name pattern or regex file name pattern and click Next.
8. Specify the polling frequency and click Next. Click on Finish to complete the adapter configuration.
9. Add a BPEL component with no service, i.e. define service later.
10. Join the wire from the FTP adapter to the BPEL component.

11. Open the BPEL component, and configure receive activity as shown below.
12. Click OK. Save and deploy the code on the Weblogic server.

Analyzing the read file as an attachment

Let's test the service and see the behaviour of file being read as an attachment.

1. Place the file in the FTP location and wait for it to get it picked.
2. Once the file is picked, log in to the em console. We will see an instance created.
3. Open the flow, here we will see only the href.
Now after seeing this, we might think about where did our data go. As we see only href. But where is our actual data?

Our data is safe within soa-infra database.

4. To query the file data and file attributes, connect to the SOA-INFRA database of the SOA instance.
5. Run the below query
select * from attachment where dbkey='href-value';
File data is stored in blob format within the attachment table.

References:


Please share your valuable feedback 😊😊😊


Comments

Post a Comment

Popular posts from this blog

DateTime formatting using xp20:format-dateTime ()

Handling XML response with or without namespace in Postman

Create Delimited String from XML Nodes and Vice Versa in SOA 12c