FTP on DecipherMiddleware

Demystifying file read as attachment in SOA 12c

Sat, 27 Nov 2021 21:12:00 +0000

In cases when we just want to move a file(without transformation) from one SFTP/FTP/file location to the other SFTP/FTP/file location, we can read a file with or without schema and then perform the write operation to the target location. If we provide an exact file schema for conversion to XML, it takes a lot of heap memory to transform from a csv or ffd to an XML document. In cases when files are not simple text files or schema is not available, one way to do so is to read the file as an opaque object and write it to a target location. In doing so, we still use a heap memory of the WebLogic server. If we are dealing with very large files, this may impact server performance as the conversion to opaque elements will take place behind the scenes. Using opaque schema is never recommended to process huge files. One better way of handling such a scenario is to read a file as an attachment in SOA 12c. Generally, we have below question:

What is reading a file as an attachment means? 🔗

In this case, we are not loading the contents of the file in the memory of the Weblogic server i.e in BPEL flow we will not see the contents of the file loaded instead, a href to the actual file is seen. In this case, the Weblogic server leverages SOA-INFRA tables to load file data in the table and in return gives href to the flow. This href actually links to data stored in the SOA-INFRA table and can be used to write files to the different locations (in the same or different FTP servers).

To understand better, let’s take an example of flow which polls the FTP directory every 10 minutes and read as an attachment. Then, how we can find the payload from SOA-INFRA tables.

Below are the steps to configure the file/FTP adapter in BPEL(SOA 12c).

Configuration Steps: 🔗

1. Create a SOA project in Jdevloper 12c.

2. Drag and drop FTP adapter from component pallet. Enter the service name. Click Next.

3. Select Define from operation and schema (specified later). Click Next.

4. Specify the source FTP JNDI name as configured in the Weblogic server. Click next.

5. Select operation type as Get File. Check the Read File As Attachment. If we know the character set, encoding, and Content-Type of the file, then specify else leave it blank. Click Next.

6. Specify the input directory from where we want to pick the file from. Check Archive processed files , and specify a directory for archive files if we want to archive the data after processing. Check Delete files after successful retrieval if we want to delete the file after processing. Click Next.

7. Provide the file name pattern or regex file name pattern and click Next.

8. Specify the polling frequency and click Next. Click on Finish to complete the adapter configuration.

9. Add a BPEL component with no service, i.e. define service later.

10. Join the wire from the FTP adapter to the BPEL component.

11. Open the BPEL component, and configure receive activity as shown below.

12. Click OK. Save and deploy the code on the Weblogic server.

Analyzing the read file as an attachment 🔗

Let’s test the service and see the behaviour of file being read as an attachment.

1. Place the file in the FTP location and wait for it to get it picked.

2. Once the file is picked, log in to the em console. We will see an instance created.

3. Open the flow, here we will see only the href.

Now after seeing this, we might think about where did our data go. As we see only href. But where is our actual data?

Our data is safe within soa-infra database.

4. To query the file data and file attributes, connect to the SOA-INFRA database of the SOA instance.

5. Run the below query

select * from attachment where dbkey='href-value';

File data is stored in blob format within the attachment table.

References: 🔗

https://docs.oracle.com/middleware/1213/adapters/develop-soa-adapters/GUID-088EFFE9-AE8B-4A35-B9DB-0A0DDA1BBBE7.htm#TKADP590

Please share your valuable feedback 😊😊😊

FTP Polling using FTP JCA adapter in OSB 12c

Thu, 14 Jan 2021 19:04:00 +0000

We can poll files at regular intervals from a remote ftp/sftp server location using Oracle JCA adapter for FTP , with help of GET operation. We can integrate oracle JCA FTP adapter with BPEL, mediator, or OSB 12c. In this article, we will be creating one OSB project that will demonstrate polling of files from remote FTP location using FTP adapter and writing the files to our local server location.

Note: Using JDeveloper 12c (12.2.1.4).

Pre-requisites: 🔗

Connection factory configured in Weblogic server.
User, configured in connection factory has the access to read and write files to target/from ftp location

Steps: 🔗

Let’s start by creating a new project with the name “FTP_Polling”.

1. Open “servicebus.sboverview” and drag FTP adapter from components palette to proxy services side.

2. Enter the adapter name and click on next.

3. In the next step, select Define from operation and schema (specified later).

4. Specify the JNDI of the connection factory(e.g. eis/Ftp/testFtpAdapter) for the remote SFTP server, from which we want to poll the files. Click on Next.

5. Specify the file type - Binary or ASCII. Select Operation type as Get File. Click Next.

6. Specify the input directory, from where we want to poll files. For e.g. “/inbound/input”. Uncheck, if you don’t want to process files recursively. If you want to delete the file from a remote location, then check “Delete files after successful retrieval”.

If you want to archive after reading, then check archive processed files. Provide the directory that is located at your server, not on the remote FTP location. If you want to archive at the same remote location, then add the below property in .jca file of the adapter.

< property name=“UseRemoteArchive” value=“true”/>

7. Specify the file pattern and click on Next.

8. Specify the polling frequency(Here we have kept 1 minute or 60 seconds. The adapter will check every 60 seconds if there is any file available) and click on Next.

9. Specify the schema, if you don’t have any schema and want to read data as opaque then select “Native format translation not required”. In this example, we will be using an opaque schema and will not convert the data to XML by providing any schema. Click on Next. Then click on Finish.

10. Adapter files along with proxy service will be created and the adapter will appear on the proxy services side.

11. Drag and Drop pipeline on Pipeline/Split joins area. A popup window will appear. Specify the service name and click on Next.

12. Uncheck the “Expose as proxy service”. Select the concrete wsdl(i.e. containing ports and bindings), which got created while creating the FTP adapter.

13. Click on Finish. A pipeline will be created. Now link the adapter and connect the arrow from proxy to pipeline as shown below.

Note: If proxy service is showing error. Just save all files in the project.

Let’s create a file adapter to write the polled file to some local directory. 🔗

14. Drag file adapter from components palette. Enter the adapter name and click Next.

15. Select Define from operation and schema (specified later). Click on Next.

16. Provide the name of JNDI and click on Next.

17. Select the Write File as operation type. Click on Next

18. Provide the physical or logical directory and file naming convention(e.g. test_%yyMMddHHmmssSS%). Click on Next.

19. Specify the schema, if you don’t have any schema and want to read data as opaque then select “Native format translation not required”. In this example, we will be using an opaque schema and will not convert the data to XML by providing any schema. Click on Next. Then click on Finish.

20. Business service and adapter files will be created. Join the wire from the pipeline to the newly created file adapter.

21. Double click on the pipeline. The pipeline window will open. You will see, the route node is already created and configured with the file write business service. Click on routing to check for properties.

22. Under properties select operation as write.

23. Since we are using opaque schema to read and write, we need not do any transformation. Opaque/ base64 data coming as input to service will go to file write business service as output. In this case, the namespace of the body variable will also remain the same.

24. Save and deploy the project to the service bus console. Check for the whole operation to execute.

Files will be picked from the source location and will be written to the target location. Sample file name: test_21011512001300

Note: Since this is an OSB composite, you need to enable trace to see the complete flow from the em console.