Batch Jobs at Different Layers in IT Environment

A batch job is an automated program or script made to conduct regular, repetitive operations automatically in software development and other business processes.

They can be programmed to run at a predetermined time or be started by an event, as when a task is finished or when data is extracted, for example.

Here are some advantages of batch jobs.

1. free up resources for other tasks by using the time and effort required to complete routine tasks.

2. tasks are more likely to be completed consistently and reliably as errors are reduced and the standard of work can be raised.

3. By handling large-scale operations, such as processing enormous amounts of data or carrying out difficult computations, can help increase the scalability and performance of business processes.

4. May lower the cost of labor or other pricey alternatives, which can be crucial for companies that need to handle a lot of data or carry out repetitive operations.

So, which layer should batch jobs be implemented at?

The application layer, database layer, and operating system layer are the three key levels of an IT environment that employ batch jobs.

1. Application Layer Batch Jobs

Batch jobs in the application layer are typically created using programming languages such as Java and Python. The great thing about application layer batch jobs is that they can be customized to meet the specific needs of your business. Examples of application layer batch jobs include data extraction and processing jobs, report generation jobs, and calculation jobs. For example, in the application layer, jobs related to business processing such as order fulfillment, inventory management, etc. can be created and executed at a certain time each day, week, or month to automate work and ensure consistency.

2. Database Layer Batch Jobs

Batch jobs in the database tier are typically created through queries or procedures and are used to perform maintenance tasks such as backups, indexing, and database optimization. In the database layer, jobs related to extracting, processing, and updating data can be created. For example, a job that extracts new data from a data warehouse every day, processes it, and updates it to a database for analysis.

3. Operating System Layer Batch Jobs

Batch jobs at the operating system layer are typically performed using shell scripts or batch files, and are used to perform system-level tasks such as backing up files, cleaning up log files, and updating the system. at the operating system layer, jobs related to server operations such as server monitoring, log management, backups, etc. can be automated.

Choosing the right layer for your batch jobs

To design an effective batch job strategy for your specific business needs, it is important to understand the characteristics of batch jobs at each layer. In general, the application layer creates batch jobs related to business processing, the database layer to extract, process, and update data, and the operating system layer to operate the server.

However, this is a universal example and does not mean that you must develop in these tiers. Choosing which tier to implement a job in depends on your organization's system goals and requirements. 

When it comes to batch jobs, the biggest confusion I hear from people I know in IT auditing is which layer should be judged to reflect batch job scheduling via continuous integration and deployment (CI/CD) tools. I typically categorize it as an application batch job because the source code is developed at the application layer and tools are used to automate the build and deployment process.

Sometimes, instead of looking up the details of a batch job in a specific tier, you might see a list of batch jobs through your organization's batch job monitoring tool, for example. 

In these cases, a quick way to figure out which tier a batch job belongs to is to first understand the nature of the batch job to make a rough guess, and then check with the appropriate person.

For example, if your batch job is primarily focused on processing data or performing application-specific tasks, it likely belongs in the Application Layer.

Or, if your batch job primarily interacts with a database, such as inserting or updating data, it likely belongs in the Database Layer.

Or if your batch job focuses on managing system resources or interacting directly with the operating system, such as managing files or processes, it's likely to be in the Operating System Layer.

Of course, it's not always clear-cut, but understanding these characteristics can help you quickly identify who is responsible during the fieldwork phase.