The InterWorks team will be providing updates to this blog as we continue to work on vulnerability remediation with our customers.
(Last Updated: 12/23/2021 3:00 PM CST)
Overview
The morning of December 10, 2021, our team became aware of a Java web server vulnerability known as log4shell that easily leads to remote code execution on the host, and thus likely loss of control of the system. This vulnerability impacts servers running the Java package log4j. This is classified as a true 10/10 risk by security organizations due to the severe impact and ease of execution. Our team will outline what we know so far and steps we are taking with our customers.
December 17th, the original fix of updating to log4j 2.15 was found to have an additional vulnerability to could still lead to remote code execution. A DDoS vulnerability was already know, but the remote code execution piece raised the CVE risk score from 3.7 to 9.0.
Scope of Vulnerability
cve-2021-44228
- Web servers running the Java package log4j.
- The vulnerability condition is disabled by default in version 2.15 or higher but still possible.
- The vulnerability condition is enabled by all versions below 2.15 and must be manually disabled in software code if not patched to version 2.15 or higher.
- log4j version 1 is expected to be vulnerable and without a patch.
cve-2021-45046
- Web servers running the java package log4j not version 2.16 or later.
Related Vulnerable Software
The following list is focused on products that InterWorks may assist you with – this is in no way a complete list of impacted services. For a better complete list, please see here: https://github.com/NCSC-NL/log4shell/tree/main/software
| Software – Version | Status Tracking / Advisory |
| Tableau Server – Any version | https://kb.tableau.com/articles/issue/Apache-Log4j2-vulnerability-Log4shell
https://status.salesforce.com/generalmessages/826 https://help.salesforce.com/s/articleView?id=000363736&type=1 |
| Matillion – Any version | https://documentation.matillion.com/docs/security-advisory-14th-december-2021 |
| VMware – Most products | https://www.vmware.com/security/advisories/VMSA-2021-0028.html |
| Dell-EMC Software/Hardware (various) | https://www.dell.com/support/kbdoc/en-us/000194414/dell-response-to-apache-log4j-remote-code-execution-vulnerability |
| Okta (RADIUS Server Agent and On-Prem MFA Agent only) | https://sec.okta.com/articles/2021/12/log4shell |
| Ruckus SZ/vSZ | https://support.ruckuswireless.com/security_bulletins/313 |
Recommended Solutions
Ultimately, the impacted software must be patched by the vendor, but alternate actions can be taken and recommended solutions are listed in order of default preference:
- Patch the vulnerable software immediately, should one exist.
- In some circumstances, software may be protected via reconfiguring the default behavior for log4j. Please engage with a service professional to evaluate if this is possible, how to do so, and to test once updated to ensure the vulnerability is truly resolved.
- Shut the service down until a patch is available.
- This is a steep measure, but the reality is this vulnerability easily leads to total loss of control to the box from a malicious third party, which would impact both it and everything around it. That would be a much greater issue than the downtime of a single service.
- Restrict the scope of access to the software – the software should be made accessible by only trusted internal business users. This does not resolve the risk, but largely reduces the scope of execution to actors inside the network.
- Protect the service via a WAF. Most products are claiming to block the vulnerability and light testing matches, but this cannot be guaranteed currently. Strong considerations should still be taken about revoking public access until patching occurs.
Notes to InterWorks Customers
InterWorks was made aware of CVE-2021-44228 vulnerability the morning of December 10th (USA) upon release and took immediate action on investigating potential impact.
InterWorks as a Vendor
A small amount of internal services were detected vulnerable and immediately had access restricted to prevent execution until patches become available. An investigation was performed during the remediation process to ensure no malicious action was taken before the point of remediation and no such indicators were found. We remain on high alert for these systems regardless.
InterWorks Products & Services
Curator – While we do use Apache for Curator, we do not use Java and therefore do not (and cannot) use Log4j, so we are not impacted by the recently discovered “Log4Shell” flaw.
Assist, ServerCare, general consulting services, etc. – None of our consulting services rely on delivery methods that are impacted by this vulnerability. While we may assist clients in remediating products that are affected, InterWorks themselves are not delivering services via methods shown to be impacted by this vulnerability.
For any further questions, please reach out to your AE, or our Information Security team.
Tableau Server Considerations
Update: 2021.12.21 11:00 AM CST
Tableau has released a second update resolving both both CVE-2021-44228 and CVE-2021-45046, this should be installed as soon as possible. Upon successful upgrade, services should be fine to return to a normal operating state.
Installs are available at: https://www.tableau.com/support/releases/server
Update: 2021.12.17 3:00 PM CST
log4j v2.15 was confirmed this morning to still permit remote code execution in certain circumstances, and must be updated to v2.16 to fix. Currently the latest versions of Tableau utilizes 2.15, therefore Tableau Server should be treated as if it were vulnerable to the latest exploits, and should maintain the same security hardening as before until a patch containing the proper version of log4j is published.
It is not yet confirmed if Tableau Server is actually vulnerable – simply that the base conditions are in place. There is currently no hotfix.
Update: 2021.12.16 1:00 PM CST
Patches and a hotfix have been published. We recommend immediately applying the available hotfix to your environment and subsequently upgrading to the latest release of your version whenever possible. Note that because of existing issues with log4j 2.15, a future upgrade will still be necessary to completely close all risks.
We still recommend our most security conscious clients keep their hotfixed or patched Tableau server away from public access, as there is a report of log4j 2.15 still being vulnerable in certain circumstances to data exfiltration. More details can be found here, but the communal understanding and impact of this is still limited so expect updates.
Key considerations:
- The hotfix only requires the downtime of stopping and starting services so it can be done quickly (15-60 minutes).
- Patches are only released for Tableau versions 2020.4 and above. Anything below this must be upgraded to be patched.
Patches: https://www.tableau.com/support/releases/server
Hotfix: https://kb.tableau.com/articles/issue/Apache-Log4j2-vulnerability-Log4shell
Update: 2021.12.15 5:00 PM CST
Tableau has published a temporary hotfix to resolve the issue. It will require a restart of services but is otherwise quick and easy to apply – one could reasonably expect 30-60 minutes of downtime depending on environment size.
This should be applied immediately! A future official patch will still be released to resolve the issue permanently.
https://kb.tableau.com/articles/issue/Apache-Log4j2-vulnerability-Log4shell
Please note, this does not resolve CVE-2021-45046, and the exploit is still under investigation to see if Tableau Server is vulnerable. As such, actions taken to isolate the server may best be left in place until a patch moving Tableau Server to log4j v2.16 is available.
Checking Server Logs
The below PowerShell script can be ran on a server or node to determine if JNDI formatted queries were sent and thus the exploit potentially tried. If results are found, they should be investigated with a security professional as additional probing will need to be performed to determine what kind of action was taken. Due to amount of obfuscation that can utilized to mask JNDI lookups a singular exhaustive search isn’t feasible. Similarly because of this a small amount of logs that are not actually JNDI executions may be flagged. Some requests may be also be present from Information Security teams testing the exploit – these should not be ignored but instead confirmed with your InfoSec team.
$output = ".\tableau-jndi-hits.log"
$logdir = $(tsm configuration get -k base.deploy.dir)+"/logs"
$logfiles = Get-ChildItem -Path $logdir -Recurse -File
foreach ($file in $logfiles) {
Select-String -Pattern '\$.*\{.*j.*n.*d.*i.*:.*\}' -Path $file -Encoding "UTF8" | Add-Content -Path $output
}
Update: 2021.12.15 4:00 PM CST
Below is a summarized overview of possible initial actions to consider while a patch is still being awaited for Tableau Server:
- The safest course of action is simply shutting down the servers down to eliminate any risk entirely. However, we know this is a large decision that takes a lot to get behind. It should be considered all the same.
- At a minimum, remove/block all inbound public or untrusted access to Tableau Server instances. Allow only inbound access from authorized users via methods such as a network firewall or VPN. This doesn’t resolve the issue but large reduces the risk front to said users.
- Outbound access does not need to be adjusted to protect from the exploit, thus things like extracts and subscriptions can still run fine. This aside, please bear in mind network security best practice in general is to restrict outbound access to only necessary destinations.
- If your IT would like to move the server to a different network to accomplish this, be sure to investigate impacts in connectivity to your IDP, data sources, email (SMTP) server, and proxy (if you have one) before hand.
- Ideally all systems are already hardened to talk to specifically what’s needed, however the truth is often not so. Work with IT to ensure security utilities like anti-virus, server and network firewalls, standard hardening practices, and monitoring are in place.
- A web application firewall (WAF) such as Cloudflare can be used to sanitize requests containing payloads, but 100% protection may not be guaranteed. Special considerations should be made when evaluating a WAF approach
- Inbound web communications to the server should only be permitted through the WAF and nothing else.
- Depending on the style of WAF, Tableau Server may need to be restarted for necessary configuration changes, resulting in brief downtime. DNS would also need to be updated which would take time to propagate.
- Tableau can transact large amounts of data through the web service such as publishing content. A WAF could impact this process due to size limitations.
- The WAF can be rolled back after the services is patched.
- Assess Tableau Server logs for attempted execution of this exploit. More details on doing so will be published shortly. Should you find any indicators of compromise, notify InfoSec and IT teams immediately.
- If you notice anything else anomalous around Tableau, contact our Information Security and IT Support teams immediately. The quicker the action, usually the smaller the scope of impact. It’s better to report a false alert than be ignorant something happened.
Update: 2021.12.15 1:15 AM CST
Salesforce has released an aggregated announcement for all products which can be found here: https://help.salesforce.com/s/articleView?language=en_US&type=1&id=000363736
At this time we are awaiting further information around Tableau Server from Tableau as specific remediation instructions have not been published. We are actively engaging with Tableau to determine such procedures.
Update: 2021.12.14 3:45 PM CST
Through testing, it has been proven that Tableau Server is exploitable through the log4j vulnerability in a manner that involves not needing to be authenticated to server. Based on this Tableau Server does need immediate attention.
While Tableau is still working on a patch, we strongly advise working with internal security resources to remove external accessibility to Tableau, either through a VPN, varying firewalls or, in major situations, shutting down Tableau Server. If this is not possible or timely, the server should be shut down.
Both these options are significant, but the vulnerability at hand easily leads to the server being compromised by virtually any external actor, which can lead to data loss and further malicious action inside the network.
As soon as a patch is available from Tableau, we will work on rolling those fixes with clients. Until Tableau publishes official fixes, the only workaround is restricting or completely removing access.
Please note that some WAF’s may not protect from this vulnerability execution on Tableau. Your own testing should be conducted post-fact.
Status can be tracked via https://status.salesforce.com/generalmessages/826.
Tableau Online Considerations
Tableau Online was notified on December 14th to be impacted by this vulnerability and actively being patched. The full scope of this impact is unknown, and unfortunately as a cloud service little action can be taken. We will advise as soon as possible once we are able to confirm details with Tableau.
Matillion Considerations
Update: 2021.12.15 1:15 AM CST
Matillion is a versatile product and as such the scope of impact is a bit more conditional. By default Matillion itself is not vulnerable in most all cases where installations at are at the latest version (1.58). As it stands only Matillion ETL for Databricks is known to be impacted, however custom configurations could also incorporate this but will need manual evaluation from members managing those items.
If you are leveraging any custom configurations, or ETL for Databricks, please work with either your staff that manages Matillion, or reach out for assistance in determining next steps.
At a minimum if a custom configuration that leverages log4j or use of Databricks is in-place, those processes should be immediately halted until patches can be put in-place.
This information is currently being tracked here: https://documentation.matillion.com/docs/security-advisory-14th-december-2021
Dell EMC Considerations
Many Dell-EMC products have been found to be vulnerable to log4shell. Workarounds and patches are being worked on and the latest status can be checked via https://www.dell.com/support/kbdoc/en-us/000194414/dell-response-to-apache-log4j-remote-code-execution-vulnerability. Should you utilize Dell EMC products in your environment, immediate review should be conducted to ensure these services are restricted from access to an acceptable degree, or taken offline.
VMware Considerations
A large variety of VMware services are impacted by this and patches are being released as soon as possible. In the interim, VMware has published config changes to workaround the issues which can be applied manually. Please do as soon as possible. Status can be tracked via https://www.vmware.com/security/advisories/VMSA-2021-0028.html.
Key Notes on CVE-2021-45046
December 14th, 2021, an additional vulnerability was found that is not resolved by either patching to log4j version 2.15 nor disabling message lookups in the utility. This vulnerability was initially not critical, but findings on December 15th by Praetorian have shown that data exfiltration is possible in some situations.
On December 17th, this report was confirmed and the CVE score was raised from 3.7 to 9.0.
Resolution
Resolving this mandates updating log4j to 2.16 which in most instances will likely be incorporated into the initial patch from vendors, or delivered shortly after as a supplemental update.
Despite v2.15 not resolving this issue, updating to it it to resolve the initial vulnerability (CVE-2021-44228) should still be done if possible to lessen possible threat vectors.
There are currently no known work-arounds to remediate this risk – patching is the only true fix.
Restricting the scope of access of vulnerable services is the best intermediate action which coincides with proper response to CVE-2021-44228.
Additional Resources
- Base Vulnerability – https://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2021-44228
- Subsequent Vulnerability – https://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2021-45046
- 2.15 Data Security Issues – https://www.praetorian.com/blog/log4j-2-15-0-stills-allows-for-exfiltration-of-sensitive-data/
- NIST Report – https://nvd.nist.gov/vuln/detail/CVE-2021-44228
- Compiled List of Impacts – https://github.com/NCSC-NL/log4shell/tree/main/software
- Sample Exploit – https://www.lunasec.io/docs/blog/log4j-zero-day/
- Sample Exploit v2 – https://www.lunasec.io/docs/blog/log4j-zero-day-severity-of-cve-2021-45046-increased/
