8.0 Troubleshooting System Updates

The following sections provide solutions to the problems you might encounter while performing a system update:

ZENworks System Update fails if Oracle DataGuard is Configured

Source: ZENworks 23.3 and later versions
Explanation: If you are using the Oracle database and configured Oracle DataGuard, while upgrading ZENworks to ZENworks 23.3 or later versions, the system update fails.
Action:

The workaround should be performed only in the following scenarios:

  • When you have configured Oracle DataGuard

  • When you are planning to upgrade to ZENworks 23.3 or later versions

  • When the System Update fails

For the ZENworks database, add the following key ”Jdbc_Url” entry in the zdm.xml file:

<entry key="Jdbc_Url">JDBC URL HERE</entry>

  • On Windows: %ZENSERVER_HOME%/conf/datamodel/zdm.xml

  • On Linux: /etc/opt/microfocus/zenworks/datamodel/zdm.xml

For the Audit database, add the following entry in the zenaudit.xml file:

<entry key="Jdbc_Url">JDBC URL HERE</entry>

  • On Windows: %ZENSERVER_HOME%/conf/datamodel/zenaudit.xml

  • On Linux: /etc/opt/microfocus/zenworks/datamodel/zenaudit.xml

NOTE:Replace the JDBC URL HERE with the actual JDBC URL. The URL can be copied from the existing JdbcUrl key if configured.

After performing the above steps, restart all the ZENworks services by running the microfocus-zenworks-configure -c Start command.

ZENworks Update fails During the Prepare state

Source: ZENworks 23.3 and ZENworks 23.4
Explanation: If you are using an SQL Server database with dynamic ports enabled and have both Port and Named Instance configured in ZENworks, then during or after the upgrade connection to the database might fail.
Action:

Perform the following steps:

  1. Remove the <Port> key entry from zdm.xml and zenaudit.xml files.

    • On Windows: %ZENSERVER_HOME%\conf\datamodel

    • On Linux: /etc/opt/microfocus/zenworks/datamodel

  2. If this key exists, remove the port from jdbc_url.

    Example: < Jdbc_Url> jdbc:sqlserver://sk-drdb01.epm.blr.novell.com:1433;databaseName=zenworksconfig;encrypt=false;instanceName=FTPDB</ Jdbc_Url>

After performing the above steps, restart all the ZENworks services using microfocus-zenworks-configure -c Start.

An error is Displayed while Logging into the ZENworks Control Center

Source: ZENworks 23.3
Explanation: While logging into the ZENworks Control Center, the following error is displayed:

500 Error Internal server error occurred. For error messages and additional information refer to API Gateway logs.

The following error is logged in the api-gateway-spring-framework.log file:

java.security.cert.CertificateException: No subject alternative DNS name matching found.

at sun.security.util.HostnameChecker.matchDNS(HostnameChecker.java:212) ~[?:?]

at sun.security.util.HostnameChecker.match(HostnameChecker.java:103) ~[?:?]

The api-gateway-spring-framework.log file is available in the following location:

  • On Linux: /var/opt/microfocus/log/zenworks/api-gateway

  • On Windows: %ZENSERVER_HOME%\logs\api-gateway

Possible Cause: This error might be logged when the hostname that is being used to connect does not match with the SAN (Subject Alternative Name) in the certificate.
Action: Remint the server certificate to include the hostname in the SAN.

System Update prepare fails while updating to ZENworks 23.3

Source: ZENworks 23.3
Explanation: While updating to ZENworks 23.3, the prepare stage might fail. Check the prepare-update.log file and the following message is logged:

"/opt/microfocus/zenworks/bin/run_preglobal_update:: OUT: Caused by: org.springframework.web.client.HttpClientErrorException$UnsupportedMediaType: 415 Unsupported Media Type: ({"timestamp":"2023-10-02T16:46:47.915+00:00","status":415,"error":"Unsupported Media Type","message":"","path":"/rest/get-zeus-version"})] [] [] [] [SystemUpdate]"

NOTE:The following workaround is applicable only if prepare fails with the above message logged in the prepare-update.log file.

Action: Manually upgrade the ZeUS RPM by running the following command as a root user and can be performed from anywhere on the server. Ensure that you run this command on all the servers where prepare fails with the above-mentioned error log message.

rpm -Uvh /var/opt/microfocus/zenworks/content-repo/system-update/5023030000fc50000000002023072812/rpm/novell-zenworks-updater-service-server-23.3.0-333.noarch.rpm

After running the command, restart ZeUS by running systemctl restart novell-zenworks-updater-service.

The Prepare stage runs every 20 minutes. Hence, after running this command, within 20 minutes the Prepare System Update stage will be re-initiated automatically.

An error message is displayed when trying to log in to ZCC after updating to ZENworks 23.3

Explanation: After updating to ZENworks 23.3, the ZCC login page might display the following error:
Possible Cause: This issue occurs when the Primary Server has multiple NICs (multiple IP addresses) and one of them is down and the ZENworks API Gateway service resolves the hostname to the IP address or NIC that is down and results in a 503 SERVICE_UNAVAILABLE error.

The below error is displayed in the log file:

[DEBUG] [2023-06-20 10:23:42] [reactor-http-epoll-6] [7] [Api-Gateway] [39] [AbstractErrorWebExceptionHandler] [[a61378b8-146] Resolved [AnnotatedConnectException: finishConnect(..) failed: No route to host: <IP address>] for HTTP POST /zenworks-location/]

[DEBUG] [2023-06-20 10:23:42] [reactor-http-epoll-6] [7] [Api-Gateway] [39] [CharSequenceEncoder] [[a61378b8-146] Writing "finishConnect(..) failed: No route to host: <IP address>"]

The log file is available in the following location:

  • Windows: %ZENSERVER_HOME%\log\zenworks\api-gateway\api-gateway-spring-framework.log

  • Linux: /var/opt/microfocus/log/zenworks/api-gateway/api-gateway-spring-framework.log

Action: Ensure the hostname resolves to a valid IP address and restart the ZENworks API Gateway service.

System Update Fails in an Embedded Postgres Zone

Source: ZENworks 2020 Update 3
Explanation: When an administrator deploys the system update to all Primary Servers at same time for an embedded Postgres zone, the system update fails on the other servers that do not have database.
Action: For an embedded Postgres zone, first update the Primary Server that has the database. Once the Primary Server with the database is updated, then update the other servers.

Update Fails on a Newly Added Server

Source: ZENworks 2020 Update 2 and ZENworks 2020 Update 3
Explanation: When you add a new Primary Server that is on ZENworks 2020 Update 2 to a ZENworks 2020 Update 3 baselined zone, the update fails on the newly added server, and the following message is logged in the system-update.log file:

Failed to download content.Content download failed for content guid

Action: Reassign the update.

Agent Update Fails during FDE Package Update

Source: ZENworks Agent, ZENworks 2020 Update 3
Possible Cause: While updating to ZENworks 2020 Update 3 on agents, if FDE and BitLocker are enabled, then the agent update fails.
Action: If FDE is enabled or the product license is active, then the BitLocker should be disabled, else the system update fails.

An HTTP 500 error is displayed when updating ZENworks

Explanation: When deploying a system update, in the Configure Update phase, an HTTP 500 error message is displayed. This error occurs if an alias hostname is used, instead of the actual DNS in the OpenID Provider.

The following is the exception error displayed when configuring an update:

Action: To resolve this issue, ensure that an actual DNS (hostname –f) is used in the OpenID Provider properties. The openid-provider.properties file is available in the following location:

  • Windows: %ZENSERVER_HOME%/conf/

  • Linux: /etc/opt/microfocus/zenworks/

Here is an example of a FQDN (fully qualified domain name) which consists of a hostname and the DNS domain name. In the following URL domain.com is a FQDN (fully qualified domain name) of a Primary Server:

openid_provider_url=https://domain.com\zenworks/?requestHandler\=ZENOpenIDHandler

Unable to Update to ZENworks 2020 Update 3, as the System Update Does not Progress

Explanation: If you have enabled ZENworks Patch Management and updating to ZENworks 2020 Update 3, the update does not progress as some of the Patch Policies have DaysToRebuild set to 0.
Action: Perform the following steps:

  1. Kill the processes that are executing the configure actions and ensure that the system update is in the failed state:

    On Linux Primary Server:

    1. Open the terminal and run the following commands:

    2. ps -aux | grep 'ConfigureLoader' After running the command, note down the Process ID (PID)

    3. kill <PID>

    On Windows Primary Server: Open the Task Manager, check and end all the running java.exe processes.

  2. After the System Update is in the failure state, run the following query to update the database:

    UPDATE zbundle SET serversidedata = replace(serversidedata, '<Variables><ns2:Name xmlns="http://novell.com/zenworks/datamodel/objects/settings" xmlns:ns2="http://novell.com/zenworks/datamodel/objects/settings">RebuildSchedule</ns2:Name><ns2:Value xmlns="http://novell.com/zenworks/datamodel/objects/settings" xmlns:ns2="http://novell.com/zenworks/datamodel/objects/settings">&lt;Schedule xmlns=&quot;http://www.novell.com/ZENworks/v1.0&quot; xmlns:xsi=&quot;http://www.w3.org/2001/XMLSchema-instance&quot; xsi:schemaLocation=&quot;http://www.novell.com/ZENworks/v1.0&quot;&gt;&lt;IntervalSchedule&gt;&lt;RepeatFrequency Months=&quot;0&quot; Weeks=&quot;0&quot; Days=&quot;0&quot; Hours=&quot;0&quot; Minutes=&quot;0&quot; Seconds=&quot;0&quot;', '<Variables><ns2:Name xmlns="http://novell.com/zenworks/datamodel/objects/settings" xmlns:ns2="http://novell.com/zenworks/datamodel/objects/settings">RebuildSchedule</ns2:Name><ns2:Value xmlns="http://novell.com/zenworks/datamodel/objects/settings" xmlns:ns2="http://novell.com/zenworks/datamodel/objects/settings">&lt;Schedule xmlns=&quot;http://www.novell.com/ZENworks/v1.0&quot; xmlns:xsi=&quot;http://www.w3.org/2001/XMLSchema-instance&quot; xsi:schemaLocation=&quot;http://www.novell.com/ZENworks/v1.0&quot;&gt;&lt;IntervalSchedule&gt;&lt;RepeatFrequency Months=&quot;0&quot; Weeks=&quot;0&quot; Days=&quot;120&quot; Hours=&quot;0&quot; Minutes=&quot;0&quot; Seconds=&quot;0&quot;') 
WHERE zuid IN (
    SELECT zuid
    FROM zzenobject
    WHERE path LIKE '%ZPM/Policy%'
      AND primarytype = 'Bundle'
      AND subtype LIKE '%Patch Bundle%'
      AND serversidedata LIKE
'%<Variables><ns2:Name xmlns="http://novell.com/zenworks/datamodel/objects/settings" xmlns:ns2="http://novell.com/zenworks/datamodel/objects/settings">RebuildSchedule</ns2:Name><ns2:Value xmlns="http://novell.com/zenworks/datamodel/objects/settings" xmlns:ns2="http://novell.com/zenworks/datamodel/objects/settings">&lt;Schedule xmlns=&quot;http://www.novell.com/ZENworks/v1.0&quot; xmlns:xsi=&quot;http://www.w3.org/2001/XMLSchema-instance&quot; xsi:schemaLocation=&quot;http://www.novell.com/ZENworks/v1.0&quot;&gt;&lt;IntervalSchedule&gt;&lt;RepeatFrequency Months=&quot;0&quot; Weeks=&quot;0&quot; Days=&quot;0&quot; Hours=&quot;0&quot; Minutes=&quot;0&quot; Seconds=&quot;0&quot;%'
                           )

    NOTE:In the above query, as an example 120 is used. However, the number can be increased if you do not want the Patch Policy to be rebuilt soon. The number 120 represents the days to rebuild the patch policy.

  3. After running the query, rerun the system update.

System Update Fails on the Linux Primary Server

Explanation: Occasionally, the system update fails on the Linux server if the ZENworks microservices are not stopped properly. To determine whether the system update failure is caused by the ZENworks microservices, execute the docker ps command. The status of the process should be displayed as actively running. Next, execute the following commands to view the logs:

  • docker exec -it zenserver /bin/bash

  • docker exec -it zenloader /bin/bash

One of the docker exe commands will not launch a container shell from the host terminal.

NOTE:You must exit the container shell, before executing the commands in the following section.

Action: Case 1: If the system update fails on SLES 15, perform the following:

  • Temporary fix: Restart the docker service by executing the systemctl restart docker.service command.

  • Permanent fix: Upgrade to SLES 15 SP2 or above and then manually update the Containerd package to version 1.3.x or above. The default Containerd package on SLES 15 SP2 is v1.2.13.

    NOTE:SLES 15 SP1 is not supported in ZENworks 2020 Update 3

Case 2: If the system update fails on SLES 15 SP2 and above, perform the following:

  • Temporary fix: Restart the docker service by executing the systemctl restart docker.service command.

  • Permanent fix: Ensure the Containerd package version 1.3.x or above is installed and restart the docker service by executing the systemctl restart docker.service command.

System Update Fails During Prepare-Update

Explanation: During the prepare-update state, the ZENworks Updater Service (ZeUS) failed to download content on a primary server
Possible Cause: This issue occurred because an attempt to download the system update was made while the server was getting restarted.
Action: Run the zac zeus-ref command to retrieve the system update or wait for the next interval at which ZENworks Updater Service (ZeUS) gets refreshed. The default refresh interval is set to 6 hours.

If the update is already configured, the application uses the previously configured port after reimporting the system update

Explanation: After configuring an update, if the system update is deleted and reimported, the application uses the previously configured port, and the status of the update is displayed as Ready to Deploy, instead of Awaiting Configuration.
Action: To modify the previously configured port, in the Configuration > System Updates page, click Action and choose Configure Update.

System update fails on a Windows 7 agent

Explanation: When system update fails on a Windows 7 agent, repair or uninstall actions cannot be performed.

Following is one of the examples of the error log message in the system-update.log file:

"[ZENUpdater] [] [SYSTEM] [SystemUpdate] [MSI_INSTALL_ERROR] [ERROR] [${content.0},1612] [] [] [ZENworks]"

Where 1612 is the MSI package is missing from the windows MSI cache.

Action: Perform the following:

  1. Identify the package or product code for which the error is displayed.

    For example, the above mentioned error 1612 is the error code for "Uninstalling {A408EF7C-6671-43EC-851A-385F1D87E847}"

  2. Run "wmic product get /format:csv > Software_%Computername%.csv"

    Open the file and search for the package or product code. The code should point to the path in the Windows, where the MSI copy should be present.

    NOTE:The actual MSI and cached MSI will have different names.

    The generic path of the file is %windir%/installer/{random_name}.msi

  3. Retrieve the corresponding package from the server. The WMI command provides the actual package name. Copy the package to the agent in the cached path, and rename the file same as the name mentioned in the WMI command, which was retrieved in Step 2.

  4. Reassign the update to the device.

After deploying the PRU, the Device status indicates that the Update has completed even for devices that are switched off, or deleted from the zone.

Source: ZENworks, Asset Management
Explanation: When you deploy the PRU (Product Recognition Update) and then check the update status, the Update Completed status is displayed even for devices that are switched off or deleted from the zone.
Action: To view the PRU status of devices:

  1. Click Bundles in ZENworks Control Center.

  2. The Bundles page is displayed, append &uid=/system to the URL of the Bundles page, system bundles are displayed.

    Example: https://ipaddress/zenworks/jsp/index.jsp?pageid=bundleList&uid=/system

  3. In the Bundles page, click the System Bundles link.

  4. In the System Bundles page, click the required Knowledge Base file.

  5. In the Bundle Status panel of the Knowledge Base page, click the here link to view the PRU system update status.

    The PRU is passed to the managed device through a bundle. The Bundle Status panel displays the device and user count against the related deployment status.

System update fails on the device

Source: ZENworks, System Update
Possible Cause: Some antiviruses may interfere with the ZENworks Endpoint Security Management installer, resulting in a system update failure of the ZENworks Agent.
Action: Refer to your antivirus documentation and make the required configuration changes to allow exclusions, prior to deploying the system update.

For more information, see TID 7007545

System update hangs

Source: ZENworks, System Update
Explanation: While importing the system update into the Zone, if the database restarts, the download progress gets stuck.
Possible Cause: During the download process, the ZENworks Loader module downloads the update and updates the database with information related to the download progress. ZENworks Control Center reads this information from the database and displays the download progress. If the database restarts in between, the communication between the ZENworks Control Center Service, the Loader module, and the database is interrupted. If the download is still in progress when the database restarts, the download status gets stuck.
Action: Cancel the download which is in progress and re-initiate it again.

OR

Delete the update and download it again.

System update of a Windows device fails because the ZENUpdater.exe executable crashes

Source: ZENworks, System Update
Explanation: During the system update of a Windows device, the ZENUpdater.exe executable crashes, and the system update fails.
Possible Cause: The ZENUpdater crashes because the .NET 4.0 framework has not been installed successfully.
Action: Verify that the .NET 4.0 framework is properly installed on the device. Re-install the .NET 4.0 framework if necessary. Restart ZENUpdater.

An administrator with System Update Deploy right and device level View Leaf right is unable to create the first stage

Source: ZENworks, System Update
Explanation: In ZENworks Control Center, the administrator with System Update Deploy right and device level View Leaf right on the entire zone is unable to create the first stage.
Action: The administrator with System Update Deploy right and device level View Leaf right on the entire zone should create a stage only after the super administrator has created the first stage.

Two ZENworks icons are displayed after System update is completed on a Mac Agent

Source: ZENworks, System Update
Explanation: After System update is completed, two ZENworks icons are displayed on a Mac agent.
Action: You must re-login or restart the device.

Unable to get updates when you perform the Check For Updates action

Source: ZENworks, System Update
Explanation: In ZENworks Control Center when you perform the Check For Updates action, there might be a failure.
Possible Cause: The Novell Customer Center (NCC) server displays a warning that the system update entitlement has expired even when that is not the case.
Action: Initiate the Check for Updates process in the background.

  • Click OK in the Retry Check for Updates dialog when you are prompted to initiate the Check For Updates process in the background. For more information, see the Section 2.2.2, Manually Checking for Updates.

    The check for updates process, which is initiated in the background, uses the values configured for the following fields in the SUEntitlementConf.properties file:

    • retryCount-CheckForUpdates

    • sleepInterval-CheckForUpdates

http-nio CLOSE_WAIT clear connection default value is set to 1 hour

Source: ZENworks, System Update
Possible Cause: On a 11 SP4 ZENworks Server, the agent ZeUS service opens a single http-nio connection with port 80. This connection will continue to be in CLOSE_WAIT state on the server even after agent ZeUS service is stopped.
Explanation: This behavior is normal as the default close time for this connection is set to a maximum of one hour. The default value of the ZeUS configurable parameter, notifier-socket-timeout, is set to 3600000. The parameter is available in the \ZeUS\Conf\zeus.conf configuration file and the default value of the parameter can be configured between 5 minutes and 1 hour.
Action: On the servers, you will see many such connections opened from multiple devices in the zone. You can ignore these CLOSE_WAIT connections as they will be cleaned up in an hour. The maximum number of connections that can be opened to this http-nio port is 20000.

Permission prompt not getting displayed on Embedded win 7 device

Source: ZENworks, System Update
Possible Cause: This is the default behavior of Windows 7 embedded device.
Explanation: The default behavior of Windows 7 embedded device could not be changed from ZENworks as MessageBox API provided by user32.dll is used.
Action: In the Windows registry, set the value of the Enabling Default Reply registry key to 0. This will disable the auto reply on prompts. The folder path of the registry key is HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Control\. For more information, refer to the MSDN solution.

ZeUS service does not working properly when a Satellite Server is demoted and then the device is updated without a reboot

Source: ZENworks, System Update
Explanation: When you demote a Satellite Server and then update the device without performing a reboot, the ZeUS service will not work properly because the rt.jar file is missing.
Action: Run the msiexec -i <filePath> TARGETDIR="<default agent installation path>" REBOOT=ReallySuppress ALLUSERS=1 /lvx*+ <log_file_path> " /qn command to generate the rt.jar file again.

In this command:

  • <filePath> is the location of the novell-zenworks-jre msi file.

  • <default agent installation path> is the location where the agent is installed.

  • <log_file_path> is the location where you want to create the log file.

Example: msiexec -i "C:\Program Files (x86)\Novell\ZENworks\cache\zmd\ZenCache\fb739230-e0de-4460-a2d2-cc1dfe1b4613\novell-zenworks-jre-1.7.0_80.x86_64.msi" TARGETDIR="C:\Program Files (x86)" REBOOT=ReallySuppress ALLUSERS=1 /lvx*+ "C:\Program Files (x86)\Novell\ZENworks\logs\system-update\5011040000fc50000000002015061004\novell-zenworks-jre-1.7.0_80.x86_64.msi.log" /qn

The ZENworks services are not restarted on SLES servers after the system update is completed

Source: ZENworks, System Update
Explanation: On SLES servers, after performing a system update the ZENworks services are not restarted automatically. Even if you manually restart the services, they stop after a couple of minutes.
Action: Restart ZeUS.

System update fails due to the lack of sufficient disk space

Source: ZENworks, System Update
Explanation: The previous zone update or upgrade has created.superceded files to ensure compatibility with older versions of the ZENworks Agent. However, these files are not required in the following scenarios:

  • The zone is baslined.

  • There are no installations of older versions of the ZENworks Agent in the zone.

The system update might fail if the disk is completely utilized by the.superceded files that are available in the following location:

  • On Window: %ZENWORKS_HOME%\install\downloads

  • On Linux: /opt/novell/zenworks/install/downloads

Action: Baseline the zone, or delete the superceded files or refer to TID 7012095 in Micro Focus knowledgebase to delete the superceded files.

Connection fails when the QuickTask was triggered on the 20.2 agent from the 20.3 primary server

Explanation: In the ZENworks 2020 Update 3, modifications were made to allow the QuickTask notification to use Apache ZooKeeper. To enable TTL-based nodes, the extendedTypesEnabled property is set to true in ZooKeeper during the system update. In a three-node ZooKeeper cluster, the extendedTypesEnabled property was enabled on a ZooKeeper node but failed on the other two nodes.

When the ZooKeeper connection was established with other nodes in the cluster, the QuickTask handler fails to create a node and the QuickTask notification also fails.

Action: Update all the nodes in the ZooKeeper cluster to ZENworks Update 3 for QuickTask to work.