Enterprise Server 3.14 release notes

A sensitive data exposure in HTML forms was possible in the management console. To mitigate this issue, the "Copy Storage Setting from Actions" functionality was removed from the management console.

MEDIUM: Malicious URLs for SVG assets provided information about a victim user who clicked the URL, allowing an attacker to retrieve metadata belonging to the user and use it to generate a convincing phishing page. This required the attacker to upload malicious SVGs and phish a victim user to click the URL for the uploaded asset. GitHub has requested CVE ID CVE-2024-9539. This vulnerability was reported via the GitHub Bug Bounty program.

HIGH: An attacker could bypass SAML single sign-on (SSO) authentication with the optional encrypted assertions feature, allowing unauthorized provisioning of users and access to the instance, by exploiting an improper verification of cryptographic signatures vulnerability in GitHub Enterprise Server. This was a regression introduced as part of follow-up remediation from CVE-2024-4985, which resulted in a new variant of the vulnerability. Please note that encrypted assertions are not enabled by default. Instances not utilizing SAML SSO, or utilizing SAML SSO authentication without encrypted assertions, are not impacted. Additionally, an attacker would require direct network access as well as a signed SAML response or metadata document. GitHub has requested CVE ID CVE-2024-9487. This vulnerability was reported via the GitHub Bug Bounty program.

A missing configuration value would cause Dependabot to be unable to create group update pull requests.

HAProxy reloading was failure prone, which could lead to failed Git operations. This reloading process has been replaced with a more resilient Systemd process.

This error message mbind: Operation not permitted was repeatedly showing in the /var/log/mysql/mysql.err MySQL logs.

The backup of audit logs could take longer after upgrading to Elasticsearch 8.

An unhandled nil value when configuring Actions storage with AWS S3 via OIDC configuration in the terminal could cause an error.

Users were unable to sign out from gist pages.

On an instance with secret scanning enabled, the custom pattern page would not load because dry run results were tied to a deleted repository.

Suspended users were not always correctly routed to the correct "suspended" page.

The "List teams" API endpoint returned duplicate results when paginating.

When managing the organization permissions required for fine-grained personal access tokens, for custom properties or projects, the Admin access level could not be selected.

A model with no URL could cause a ghe-migrator import to fail.

The ghe-spokesctl status command showed repaired repositories as broken if their network ID changed during the repair (for example, when the repository was detached from it's original network).

Missing URLs on import could lead to migration failures without logging or explanation.

On the security overview dashboard, data initialization could fail when creating new organizations or changing GitHub Advanced Security licensing.

Restore could fail when restoring MySQL using backup-utils.

ghe-remove-node will display the log file location when running in quiet mode.

Pre-receive hook environments can use the clone3() system call.

The creation, deletion, or change in visibility of a gist has been added to the audit log.

During the validation phase of a configuration run, a No such object error may occur for the Notebook and Viewscreen services. This error can be ignored as the services should still correctly start.

If the root site administrator is locked out of the Management Console after failed login attempts, the account does not unlock automatically after the defined lockout time. Someone with administrative SSH access to the instance must unlock the account using the administrative shell. For more information, see "Troubleshooting access to the Management Console."

On an instance with the HTTP X-Forwarded-For header configured for use behind a load balancer, all client IP addresses in the instance's audit log erroneously appear as 127.0.0.1.

In some situations, large .adoc files stored in a repository do not render properly in the web UI. The raw contents are still available to view as plaintext.

Repositories originally imported using ghe-migrator will not correctly track Advanced Security contributions.

The admin stats REST API endpoints may time out on appliances with many users or repositories. Retrying the request until data is returned is advised.

When following the steps for Replacing the primary MySQL node, step 14 (running ghe-cluster-config-apply) might fail with errors. If this occurs, re-running ghe-cluster-config-apply is expected to succeed.

Running a config apply as part of the steps for Replacing a node in an emergency may fail with errors if the node being replaced is still reachable. If this occurs, shutdown the node and repeat the steps.

If a hotpatch upgrade requires the haproxy-frontend service to be restarted, the restart will hang if there are existing long-lived connections, such as browser web sockets or Git operations. No new connections will be accepted for up to 5 minutes. Any existing unfinished connections at this time will be disconnected.

When restoring data originally backed up from a 3.13 appliance onto a 3.13 appliance, the elasticsearch indices need to be reindexed before some of the data will show up. This happens via a nightly scheduled job. It can also be forced by running /usr/local/share/enterprise/ghe-es-search-repair.

An organization-level code scanning configuration page is displayed on instances that do not use GitHub Advanced Security or code scanning.

In the header bar displayed to site administrators, some icons are not available.

When enabling automatic update checks for the first time in the Management Console, the status is not dynamically reflected until the "Updates" page is reloaded.

When restoring from a backup snapshot, a large number of mapper_parsing_exception errors may be displayed.

Services may respond with a 503 status due to an out of date haproxy configuration. This can usually be resolved with a ghe-config-apply run.

Images embedded in wiki pages may stop rendering shortly after being published. [Updated: 2024-10-16]

MEDIUM: An attacker could steal sensitive information by exploiting a Cross-Site Scripting vulnerability in the repository transfer feature. This exploitation would require social engineering. GitHub has requested CVE ID CVE-2024-8770 for this vulnerability, which was reported via the GitHub Bug Bounty program.

On an instance with GitHub Actions enabled, due to an insufficient wait time, MS SQL and MySQL replication could fail with the error message Failed to start nomad service!.

ghe-storage-find was sometimes unable to identify a data disk.

After upgrading the relevant GHES version, the resolvconf service failed to start due to a missing directory.

Some pre-receive hooks using the faccessat2 system call, such as those using Alpine Linux as the base, failed unexpectedly.

When configuring a high availability replica and during the database seeding of a MySQL replica node, restarting the nomad service could time out. Consequently, when MySQL replication attempted to start an error was reported, and setting up replication failed.

On an instance in a cluster configuration, the ghe-cluster-status command returned an error if a soft-deleted repository had a checksum mismatch.

Fixes and improvements for the git core module.

Some repositories could miss spokes information after restoring in a clustering topology due to unrescued exceptions.

In organizations with a large number of repositories, when an administrator used repository properties to target repositories in an organization ruleset, the ruleset index page timed out.

After a user created a Projects Insights chart with time as the X-axis, the chart became hidden and inaccessible.

Fixes a known issue where some links to GitHub Docs from GitHub Enterprise Server may lead to a “Page not found.” Previously, the links incorrectly added enterprise-cloud@latest to the URL.

A bug introduced in 3.12 which prevented the search input in the global navigation from displaying a dropdown of search suggestions has been fixed. The search input functionality prior to 3.12 has been restored, and users are once again able to see and submit suggested search queries, including scope suggestions.

Custom links to other repositories displayed incorrect breadcrumbs.

The Secret Scanning Push Protection custom resource link set at the Enterprise level was not being displayed to users being blocked when pushing secrets to a repository using git through the command line interface.

Following an upgrade, Elasticsearch search migrations are sometimes incorrectly reported as failing in the audit log, even though the migrations completed successfully. [Updated: 2024-09-27]

For instances deployed on Amazon Web Services (AWS), site administrators can configure regional AWS STS endpoints for OIDC from the Management Console.

Site administrators can now configure the instance with NUMA optimizations.

During the validation phase of a configuration run, a No such object error may occur for the Notebook and Viewscreen services. This error can be ignored as the services should still correctly start.

If the root site administrator is locked out of the Management Console after failed login attempts, the account does not unlock automatically after the defined lockout time. Someone with administrative SSH access to the instance must unlock the account using the administrative shell. For more information, see "Troubleshooting access to the Management Console."

On an instance with the HTTP X-Forwarded-For header configured for use behind a load balancer, all client IP addresses in the instance's audit log erroneously appear as 127.0.0.1.

In some situations, large .adoc files stored in a repository do not render properly in the web UI. The raw contents are still available to view as plaintext.

Repositories originally imported using ghe-migrator will not correctly track Advanced Security contributions.

Admin stats REST API endpoints may timeout on appliances with many users or repositories. Retrying the request until data is returned is advised.

When following the steps for Replacing the primary MySQL node, step 14 (running ghe-cluster-config-apply) might fail with errors. If this occurs, re-running ghe-cluster-config-apply is expected to succeed.

Running a config apply as part of the steps for Replacing a node in an emergency may fail with errors if the node being replaced is still reachable. If this occurs, shutdown the node and repeat the steps.

If a hotpatch upgrade requires the haproxy-frontend service to be restarted, the restart will hang if there are existing long-lived connections, such as browser web sockets or Git operations. No new connections will be accepted for up to 5 minutes. Any existing unfinished connections at this time will be disconnected.

When restoring data originally backed up from a 3.13 appliance onto a 3.13 appliance, the elasticsearch indices need to be reindexed before some of the data will show up. This happens via a nightly scheduled job. It can also be forced by running /usr/local/share/enterprise/ghe-es-search-repair.

An organization-level code scanning configuration page is displayed on instances that do not use GitHub Advanced Security or code scanning.

When following the steps for Replacing the primary MySQL node, step 14 (running ghe-cluster-config-apply) might fail with errors. If this occurs, re-running ghe-cluster-config-apply is expected to succeed.

In the header bar displayed to site administrators, some icons are not available.

When enabling automatic update checks for the first time in the Management Console, the status is not dynamically reflected until the "Updates" page is reloaded.

When restoring from a backup snapshot, a large number of mapper_parsing_exception errors may be displayed.

Services may respond with a 503 status due to an out of date haproxy configuration. This can usually be resolved with a ghe-config-apply run.

Images embedded in wiki pages may stop rendering shortly after being published. [Updated: 2024-10-16]

Instance administration

• On an instance with multiple replica nodes, to start or stop replication for all nodes in a single configuration run, administrators can use the ghe-repl-start-all and ghe-repl-stop-all commands.

Instance services

• Administrators can scale the appliance using generation 2 virtual machines, with support for booting in UEFI mode. This requires deploying a new instance and restoring data onto it. See "Using generation 2 virtual machines."

• Nomad has been upgraded to 1.5.17 and Consul has been upgraded to 1.17.4. These services are used in GitHub Enterprise Server to orchestrate containers and configuration.

Identity and access management

• Automated user provisioning via the System for Cross-domain Identity Management (SCIM) standard is available in public beta. Instances that use SAML authentication can enable SCIM to provision user accounts and manage their lifecycle from an identity provider (IdP). You can configure SCIM using an application for supported IdPs, or using the REST API endpoints for SCIM. See "Configuring user provisioning with SCIM on GitHub Enterprise Server."

  • If your instance already uses SAML, you will need to configure a new IdP application that supports automated provisioning via SCIM.
  • Existing private beta customers should also reconfigure their implementation with an updated application.
  • During the public beta, we recommend testing SCIM support for your identity system in a non-production instance before adding SCIM to your current setup.

• Organization owners can create and assign custom organization roles, delegating administrative duties to trusted teams and users. See "Managing custom organization roles."

• Users can use the account switcher to switch between multiple accounts. See "Switching between accounts."

• On an instance that uses built-in authentication, users can use passkeys to sign in securely to GitHub, without needing to input their password. See "Authenticating with a passkey."

• Enterprises that use an SSH certificate authority can allow SSH certificates to be used to access user-owned repositories. See "Enforcing policies for security settings in your enterprise."

Audit logs

• Every 24 hours, a health check runs for each audit log stream. If a stream is set up incorrectly, an email will be sent to the enterprise owners as notification that their audit log stream is not properly configured.

Secret scanning

• Users can specify which teams or roles have the ability to bypass push protection. This feature is in public beta and subject to change. See "About push protection."

• Secret scanning detects secrets leaked in discussions and in pull request titles, bodies, and comments. This feature is in public beta and subject to change. See "About secret scanning."

• Secret scanning blocks contributors from uploading files with detected secrets if push protection is enabled for a repository. This feature is in public beta and subject to change.

• Audit log events are created when secret scanning non-provider patterns are enabled or disabled at the repository, organization, or enterprise level.

Code scanning

• Users can create a dedicated code scanning rule to block pull request merges, instead of relying on status checks. This feature is in public beta and subject to change. See "Set code scanning merge protection."

• Users can use CodeQL threat model settings for C# to adapt CodeQL's code scanning analysis to detect the most relevant security vulnerabilities in their code. This feature is in public beta and subject to change. See "Editing your configuration of default setup."

• Organizations that use default setup for code scanning can use organization-level model packs to extend the coverage of multiple repositories. This feature is in public beta and subject to change. See "Editing your configuration of default setup."

• CodeQL can scan Java projects without a build. This feature is in public beta and subject to change.

• This release comes installed with version 2.17.6 of the CodeQL CLI, used in the CodeQL action for code scanning. Significant updates since the default version installed on GitHub Enterprise Server 3.13 include:

  • Support for Java 22, Swift 5.10, TS 5.4, and C# 12
  • New queries for C/C++, Go, Java, and Ruby:
    • cpp/type-confusion: Detects casts to invalid types
    • cpp/iterator-to-expired-container: Detects the creation of iterators owned by temporary objects that are about to be destroyed
    • go/uncontrolled-allocation-size: Detects slice memory allocation with excessive size value
    • java/unvalidated-url-forward: Prevents information disclosure caused by unsafe URL construction
    • rb/insecure-mass-assignment: Detects instances of mass assignment operations accepting arbitrary parameters
    • rb/csrf-protection-not-enabled: Detects cases where Cross-Site Request Forgery protection is not enabled in Ruby on Rails controllers

Dependabot

• Users can consolidate Dependabot pull requests by enabling grouped security updates for related dependencies in a package ecosystem. See "About Dependabot security updates."

• Dependabot can access Cargo private registries to provide updates to Rust dependencies. See "Guidance for the configuration of private registries for Dependabot."

• Dependabot pauses scheduled jobs after 15 failures. This gives an earlier indication of potential issues while still ensuring that critical security updates continue to be applied without interruption.

• Dependabot uses private registry configurations specified in the dependabot.yml file as expected, even if there is a configuration with target-branch. This ensures that security updates are applied correctly, regardless of your repository's configuration settings. See "Configuring access to private registries for Dependabot."

• In the dependabot.yml file, users can apply the same configuration to manifest files from multiple directories using the directories key. Direct strings, glob syntax, and wildcards (*) are all supported for targeting directories. See "Configuration options for the dependabot.yml file." [Updated: 2024-10-07]

Code security

• The security overview dashboard, with the ability to view secret scanning metrics and trending data for the enablement of security features, is available at the enterprise level. See "Viewing security insights."

• The security overview dashboard for organizations is now generally available.

• On the security overview dashboard, users can view alert trends grouped by tool. The group-by option is designed to improve the ability to track and analyze the effectiveness of scanning tools, enabling more strategic decision-making. See "Viewing security insights."

• On the security overview dashboard, users can filter by security tool. This feature is in public beta and subject to change.

• In the dependency graph, a software bill of materials (SBOM) generated for a package now includes the package URL for more packages. Previously, the package URL was not included if the manifest file referenced a package with a version range.

GitHub Actions

• For self-hosted GitHub Actions runners on this GitHub Enterprise Server release, the minimum required version of the GitHub Actions Runner application is 2.317.0. See the release notes for this version in the actions/runner repository. If your instance uses ephemeral self-hosted runners and you've disabled automatic updates, you must upgrade your runners to this version of the Runner application before upgrading your instance to this GitHub Enterprise Server release.

• Deployment views across environments are now generally available. Users can pin environments and use additional filters to filter the views. See "Viewing deployment history."

GitHub Pages

• Users can configure custom GitHub Actions workflows to build and deploy sites on GitHub Pages. See "Configuring a publishing source for your GitHub Pages site."

Repositories

• Users can enhance security by adding deploy keys as a bypass type to rulesets. See "Creating rulesets for a repository."

• Users can select Dependabot in the bypass list of a ruleset. See "Creating rulesets for a repository."

Projects

• Users can use the auto-close issue workflow to automatically close issues when a project item moves to a specific "completed" status. See "Using the built-in automations."

Integrations and extensions

• When authenticating to a native GitHub App or OAuth app, users will be prompted to select which account they want to sign in to using an account picker. Developers of apps can append ?prompt=select_account to their login flow to show users the account picker.

• When using a JSON Web Token (JWT) to authenticate or request an installation token, developers of GitHub Apps can use the app's client ID for the JWT's iss claim. The application ID remains valid, but is considered deprecated.

Pushes that update over 5,000 branches no longer trigger webhooks or GitHub Actions workflows.

Complete SCIM payloads are written to the audit log, including SCIM attributes that are not required or supported per API docs. Customers using Okta with SCIM may notice that a placeholder password attribute is among the data passed to audit logs in its current configuration. This placeholder data is associated with Okta’s password synchronization feature that is not expected or required by GitHub. See okta-scim for more information.

Custom firewall rules are removed during the upgrade process.

When enabling automatic update checks for the first time in the Management Console, the status is not dynamically reflected until the "Updates" page is reloaded.

During the validation phase of a configuration run, a No such object error may occur for the Notebook and Viewscreen services. This error can be ignored as the services should still correctly start.

If the root site administrator is locked out of the Management Console after failed login attempts, the account does not unlock automatically after the defined lockout time. Someone with administrative SSH access to the instance must unlock the account using the administrative shell. For more information, see "Troubleshooting access to the Management Console."

On an instance with the HTTP X-Forwarded-For header configured for use behind a load balancer, all client IP addresses in the instance's audit log erroneously appear as 127.0.0.1.

In some situations, large .adoc files stored in a repository do not render properly in the web UI. The raw contents are still available to view as plaintext.

Repositories originally imported using ghe-migrator will not correctly track Advanced Security contributions.

REST API endpoints for admin stats may time out on appliances with many users or repositories. Retrying the request until data is returned is advised.

When following the steps for Replacing the primary MySQL node, step 14 (running ghe-cluster-config-apply) might fail with errors. If this occurs, re-running ghe-cluster-config-apply is expected to succeed.

Running a config apply as part of the steps for Replacing a node in an emergency may fail with errors if the node being replaced is still reachable. If this occurs, shut down the node and repeat the steps.

If a hotpatch upgrade requires the haproxy-frontend service to be restarted, the restart will hang if there are existing long-lived connections, such as browser web sockets or Git operations. No new connections will be accepted for up to 5 minutes. Any existing unfinished connections at this time will be disconnected.

When restoring data originally backed up from a 3.13 appliance, the Elasticsearch indices need to be reindexed before some of the data will show up. This happens via a nightly scheduled job. It can also be forced by running /usr/local/share/enterprise/ghe-es-search-repair.

The global search bar does not have suggestions enabled due to the redesigned navigation and pending new search experience.

Upgrades include an error concerning Error deregistering job for consul-template. This message does not indicate any problems with your install and can be safely ignored.

Some links to GitHub Docs from GitHub Enterprise Server may lead to a "Page not found," because an enterprise-cloud@latest portion is incorrectly added to the URL.

An organization-level code scanning configuration page is displayed on instances that do not use GitHub Advanced Security or code scanning.

In the header bar displayed to site administrators, some icons are not available.

When restoring from a backup snapshot, a large number of mapper_parsing_exception errors may be displayed.

On boot, the resolvconf service may fail to start because the /run/resolvconf directory does not exist when the service attempts to touch a file there, with the error:

/bin/touch: cannot touch '/run/resolvconf/postponed-update': No such file or directory

If this occurs, workaround this issue with the following commands — this change will persist on reboots, but not upgrades:

sudo sed -i.bak \
'/\[Service\]/a ExecStartPre\=\/bin\/mkdir \-p \/run\/resolvconf' \
/etc/systemd/system/resolvconf.service.d/local.conf

sudo systemctl daemon-reload
sudo systemctl start resolvconf 

Services may respond with a 503 status due to an out of date haproxy configuration. This can usually be resolved with a ghe-config-apply run.

When enabling automatic update checks for the first time in the Management Console, the status is not dynamically reflected until the "Updates" page is reloaded.

Following an upgrade, Elasticsearch search migrations are sometimes incorrectly reported as failing in the audit log, even though the migrations completed successfully. [Updated: 2024-09-27]

Images embedded in wiki pages may stop rendering shortly after being published. [Updated: 2024-10-16]

The Manage GHES API reached feature parity with the Management Console API in GHES 3.12. As a result, we will remove the Management Console API in GitHub Enterprise Server 3.15. For information about updating tooling that relies on the Management Console API, see "REST API endpoints for Management Console."

These release notes previously indicated as a known issue that on GitHub Enterprise Server 3.14.0 when log forwarding is enabled, some forwarded log entries may be duplicated. The fix for this problem was already included prior to the release of GitHub Enterprise Server 3.14.0. [Updated: 2024-09-16]

These release notes did not include a note for support of the directories key in dependabot.yml files. [Updated: 2024-10-07]