I’ve been pinged on several occasions to help out with issues with the Cloud File Transfer Agent (CFT Agent) over the last six months so I thought I’d pen an article around the CFT Agent, troubleshooting, observations and I’ll throw in some anecdotes to boot 🙂

The CFT Agent

The CFT Agent one of the ‘on premise’ components that we need to install as part of our M3CE installs. When you get that ‘wait’ status when going in to the Business Engine Files or Business Engine Data Management under the Administration Tools, this is checking to make sure that there is communication with the CFT Agent.

The CFT Agent is pretty nifty, it will establish an outbound https connection to M3CE and work passes over the connection it creates. Very handy as it means that we don’t need to worry about port forwarding and exposing our infrastructure to the internet and yet M3CE can still send data to the CFT Agent. We just need to allow the server to communicate out over https to the cloud.

So what does it actually do?

When you do a CMS005 -> Related Options -> Export or one of the many other programs that allow us to export, it will generate a data file which gets transferred down to the CFT Agent, the import gets read from the CFT Agent. Equally, when you do a company copy, once the data is extracted and compressed, it is sent down to the CFT Agent – for the eagle eyed, you’ll probably notice that there is the option to do copies directly between tenants now. Some of the finance functions will also use the grids MvxFileTransfer, as does EVS100.

If you have turned on the save option in MNS204 for a document, and you do a MNS270 -> Related Options -> View, then it will query the CFT Agent to see the file exists on the CFT Agent. It’ll retrieve that file and display it for you. If your CFT Agent is down, then you won’t be able to display the file. If the CFT Agent is down and you are generating print files, then (assuming it hasn’t changed recently) once the CFT Agent starts up, those files will be transferred down to the CFT Agent…this of-course means that if it hasn’t been running for several months (like one of our sandboxes) it took ages for the CFT Agent to catch up when I had to install it on my local computer.  We really should make sure it is running for each tenant.

M3CE only supports ONE CFT Agent per tenant. If you install a second CFT Agent for the same tenant, then expect issues where files ‘disappear’ as the agent retrieves a file list from one CFT Agent when some of the files are on the other. Don’t do it.

The CFT Agent itself is installed in to the grid – those of us that worked with 13.x will recall the grid leverages Jetty, a Java Webserver and Servlet. The CFT Agent itself ends up being an application within the grid. The CFT agent grid doesn’t really seem as heavy as the grids of old, infact the installation of the CFT Agent itself really feels fairly lightweight and ‘appliance based’. Often if something goes wrong (like people removing a JDK, upgrading to a different JDK, nuking the database server – yes, I’ve come across all of these :-)) then may be just as easy to download the CFT Agent and install it fresh.

Installation

Below is how I tend to like installing the CFT Agent. There are different thoughts, philosophies and approaches – there’s no “one right way”, but there are certainly ways to make your life easier (and there’s definitely wrong ways :D). Below I’ll discuss my general preferences – though it is important to recognise that there are multiple considerations and factors involved in how you deploy your CFT Agents.

My personal preference is a separate server for each tenant, but the reality is that people often don’t want the licensing costs associated with three servers for something that doesn’t really do much. Often what will happen is the CFT Agent for two non-production tenants will be on one server, and the PRD tenants CFT Agent will be on its own. It’s not uncommon for Enterprise Connectors for the non-production tenants to also end up being installed on the same server as the non-production CFT Agent – again, it’s not my preference but it is an approach I commonly see.

JDK – Corretto

I tend to prefer installing in to the default location. I know some people that will install the JDK in to specific directories. The concern that I tend to have here is that from an administration perspective we can end up with JDKs all over the place. It makes it more difficult from an exceptions policy process when you have exceptions for unusual places which can be overlooked by the administration staff.

The CFT Directory

I tend generally make sure that I drop the CFT Agent in to a directory where we can easily identify which tenant type (PRD, TRN, DEM, DEV etc) the CFT Agent is for and also identify the grid is for a CFT Agent. Equally, I like to install the grid in to the CFT Agent directory and I will make sure that the grids directory name identifies the tenant and that it is a CFT Agent.

The reason for this is that the Grid directory name influences the security group that is created and the service account name that is created. It also makes it easier when we troubleshoot because we can see paths that identify the tenant and agent. This is really only an issue if you have more than one CFT Agent on a server or you run the CFT Agent and the Enterprise Connector, but I consider it a good practice.

So my normal installs will look like this – my CFT directory is <tenant>_<tenant type>_CFT sits under a the directory Infor. You can see below that the grid directory is InforIONGridAX2CFT.

You can see that the security group corresponds to the grid directory name. I usually add myself to that group pretty early on otherwise we won’t be able to see the contents of some of the directories.

And it makes it easier visually when looking at the processes, we can see there are multiple CFT Agents running on this server and at a glance we can see which process is associated with which CFT Agent.

The directory name also influences the Windows service name.

M3PRINTS – M3 Print Files

If the MNS204, Save flag is turned on, then when we generate a standard output, be it a document or a report, it will be saved to the M3PRINTS directory. The .xml files will typically end up generating a .pdf, the .csv will typically be transformed in to a .xlsx file.

M3FILES – MvxFileTransfer, Config Files etc

MvxFileTransfer

This directory is where the BEDataMgmt, Config Data, File Import directories live. It is also the directory where several of the financial functions pick up files.

BEDataMgmt

When you do company copies, this is the directory where the .zip files get deposited. Yes, you can copy these files directly from the between tenants CFT Agents if you have access to the servers file system. Normally I like to use a specific naming convention when doing an export so it makes files easier to identify. My convention is

❤ letter tenant id>_<Component>_<Company>_<Division or All>_<Table or All>Tables.

An example would be PRD_MVX_100_All_AllTables.zip, which would be from the PRD tenant, MVX component, company 100, all divisions, all tables.

Config Data

This is where files go when you select the Export option from various programs (eg. CMS005, CMS045, CRS021 etc).

File Import

This is the directory where the files read by EVS100 live.

General House Keeping

The CFT Agent itself, will upgrade, however the grid currently won’t. And given that there is usually a new grid released each month, it’s a good idea to keep up to date.

I had a situation in the early days of M3CE when it was still under controlled availability, before anyone had gone live, where a customer was having all sorts of timeouts in the UI when they were using some programs that generated outputs. It was absolutely baffling. It turned out that the issue was related to the CFT Agent being updated, however it required functionality in a newer version of the grid, when trying to interact with the print outputs, errors would be generated. This in turn created an unusual situation where M3CE was waiting for the stream to be sent and the UI would timeout.
An upgrade to the grid resolved the situation and the latent behaviour with the timeout was addressed – normally you wouldn’t see any delays in the UI even if the CFT Agent is down.

I tend to recommend that people make sure that their non-production CFT Agents be kept up to date. So once a month, shortly after the maintenance window, upgrade those CFT Agents, one at a time. Then after a couple of weeks, once you’re happy, upgrade the CFT Agent in your prod tenant.

Also, because files get stored on the server

• Clean up print files (I like to schedule the print clean up job in the M3 Business Engine)
• Clean up configurations
• Clean up data files

The reason that I tend to recommend this is because some file systems start to choke when there are large numbers of files in a directory. Normally on Windows you often notice slow downs when you have more than 10k files, definitely over 100k files. And don’t underestimate how quickly print files can accumulate.

Upgrades

As I mentioned before, the CFT Agent Client itself auto upgrades by default, because the CFT Agent itself depends on the grid, we really should make sure that we keep the grid updated too. These upgrades usually become available just after the monthly maintenance window, though I have seen grid upgrades released outside of the maintenance window.

My general recommendation is that upgrade your least critical CFT Agent first, ensure that the upgrade works as expected. Then upgrade your next non-production CFT Agent. Let them run for a week or two and once you’re happy, upgrade your production tenants grid.

How do you upgrade?

Navigate to your grid installation directory -> grid -> applications -> CloudFileTransferClient -> appdata -> installers

You’ll see various .cmd files, these are the files that you want to run. Make sure that you run the latest and run as an Administrator. The version has the bundle date and time after the version.

Upgrading Java

Be cautious when upgrading the JDK – make sure you are only use the supported version, and if you are upgrading, install your updated JDK, run the ChangeJDK tool in the grid for each of your grids on that server, make sure they still run and then uninstall the old version.  I would also make sure that you upgrade a non-production first, let it run for a while before upgrading the JDK for your production system.

Bin Files and Tools

The AdminUI and the AdminUIClassic can be used for examining the state of the CFT Agent. My preference has been to use the AdminUIClassic, however it doesn’t have all the functionality of the newer AdminUI, so you’re probably better off using the AdminUI.

They offer you the opportunity to view the logs and verify if the application states of healthy.

ChangeJDK – this batch file will allow us to change the JDK being used. Essentially this will change the path in the batch files and in the grids config\bootstrap.properties file. I have had situations where we have to manually edit the ChangeJDK batch file so it’s path points to a valid JDK.

OfflineConfigUI allows us to edit some of the settings of the grid when it is offline. I tend to feel that reinstalling the CFT Agent is probably a better option if you get to this point. I’ve only used it once and that was because I had very limited access to the SQL Server.

StartAllHosts, StartHost, StopAllHosts, StopHost. This will stop/start the hosts as appropriate.

The batch files above, tend to leverage .jar files in the tools directory. One of the ones that isn’t is the grid-cli.jar. With the grid-cli you can issue commands to the grid, this allows you to do things like change the password to your database server. Take a look at the Grid Administration Guide in the support portal, it has a lot of good information.

Troubleshooting

There’s no single approach to troubleshooting, but when I am looking at issues with the grid, one of the first things is to check what processes are running. We can do this by opening up Task Manager, right clicking on the name column and add the Command line.

I normally drag the column over to be the right-most column and expand it out as far as possible so we can see the command line that launched the process.

Here I can see the path to our JDK, when using my naming convention I can see that I have the right CFT Agent.

Each of the applications within the grid will be run as its own process. Each of the routers within the grid will also be run in their own processes. From the screenshot above, we can see that a CFT Agent runs two processes here. The Host Router and the CFTClient. If you were looking at the Enterprise Connector, you’d potentially see three processes – a host router, the Enterprise Connector and Enterprise Print.

Check the Admin UI and check the state of the grid application. The notifications should give you some useful information, and you can quickly and easily see the host state and the state of the applications in the grid.

You can take a look at the applications to see their state directly, and you can also see if there are errors or warnings.

You can also check the logs to see if there is any useful information.

When looking at the logs, you can also look at the file system. Sometimes it can be a little confusing as to what logs are containing what information.

The log files in the <grid install directory>\log relate specifically to the grid. Usually if I am having issues with the grid starting, I’ll look here. If the grid starts and the application starts (even if it can’t connect), usually we don’t need to look to hard in this directory.

The bootstrap is usually a good place to look if the host router doesn’t start. (The bootstrap is normally started by the windows service)

Incidentally, you can double click on the .html file in the grid root directory to see the state of the bootstrap

And it will look something like this – it can be a good source of information.

For the host router and the actual CFT Agent logs, you would look at the log directory under the grid

Under the log we can see the CloudFileTransferClients log directory – check these if the grid is running ok but you can’t connect to M3CE

Issues that I’ve seen

These are some of the issues that I’ve come across. As I mentioned previously, I tend to work on the basis that if it is taking too long to troubleshoot and it’s obvious the problem is the grid or the CFT Agent, I’ll normally reinstall. The reinstall process is usually pretty painless.

Firewall being too Restrictive

I came across a scenario where the firewall was a little too restrictive so the CFT Agent couldn’t establish a connection to the cloud where M3CE runs. Once the restrictions were reduced, the CFT Agent could communicate out.

Not Starting

Though it’s rare, and it’s usually due to other external reasons, often the reason that the grid doesn’t start is because the grid didn’t shut down properly. Normally, I’ll run the stop the hosts, then I’ll take a look at Task Manager to see if there is still a job, if after a while it isn’t stopping I’ll kill the task and then start the grid.

If you’ve changed the service account user, make sure that the password is correct.

Compliance systems / Anti-Virus

I had one situation when during a project I got asked to look at a CFT Agent that was running however it was generating hundreds of errors every minute. As it turned out, they had just installed a ‘compliance’ system for their servers. Because Java hadn’t been added to the approved list of the compliance system for the server, the system uninstalled the JDK – however, because some files were in use, it didn’t uninstall completely which meant that the grid was running – to a point, but wouldn’t work properly.

JDK Updates / auto-patching systems

I hit multiple situations where customers have clicked on the “Upgrade” when Java prompts you to, this has ended up uninstalling the old version and installing the new version – the path has changed which means that the grid will not run. I’ve also seen people updating from the JDK 1.8 to JDK 11, at the time of writing, the CFT Agent isn’t happy trying to run under different versions of Java.

It’s also worth noting, Corretto is the recommended JDK to use.

I hope that this provides some insight in to the CFT and where to start looking if you have issues.