Enterprise Server 3.10 release notes

MEDIUM: An attacker could steal sensitive information by exploiting a Cross-Site Scripting vulnerability in the repository transfer feature. This exploitation would require social engineering. GitHub has requested CVE ID CVE-2024-8770 for this vulnerability, which was reported via the GitHub Bug Bounty program.

MEDIUM: An attacker could push a commit with changes to a workflow using a PAT or OAuth app that lacks the appropriate workflow scope by pushing a triple-nested tag pointing at the associated commit. GitHub has requested CVE ID CVE-2024-8263 for this vulnerability, which was reported via the GitHub Bug Bounty program.

For instances deployed on AWS with IMDSv2 enforced, fallback to private IPs was not successful.

A config apply run may not have been properly applied due to calls being made to Nomad before it was ready to accept connections. When this occurred, the Error querying agent info: failed querying self endpoint: Get "http://127.0.0.1:4646/v1/agent/self" error was written to the /data/user/common/ghe-config.log file.

When configuring a high availability replica and during the database seeding of a MySQL replica node, restarting the nomad service could time out. Consequently, when MySQL replication attempted to start an error was reported, and setting up replication failed.

When importing using ghe-migrator, team URLs containing dots were imported as-is, leading to 404s when attempting to view the imported teams. Dots in imported team URLs are now escaped to dashes.

On an instance in a cluster configuration, the ghe-cluster-status command returned an error if a soft-deleted repository had a checksum mismatch.

Some repositories could miss spokes information after restoring in a clustering topology due to unrescued exceptions.

Fixes and improvements for the git core module.

The CommandPalette component no longer displays repository information on 404 pages, preventing the leakage of private repository information for users without access.

Custom links to other repositories displayed incorrect breadcrumbs.

When a GitHub App installation had all repositories installed individually, it was not possible to remove the repositories from the selection.

After an administrator enabled maintenance mode from an instance's Management Console UI using Firefox, the administrator was redirected to the Settings page, but maintenance mode was not enabled.

Some custom pattern matches were incorrectly filtered during post-scan filtering. You may want to edit and republish your custom patterns. You can manually republish custom patterns with the following command: ghe-secret-scanning jobs queue custom-patterns republish --custom-pattern-id=?. Outdated alerts caused by edits during custom pattern backfills have been fixed in version 3.13 and above.

For instances deployed on Amazon Web Services (AWS), site administrators can configure regional AWS STS endpoints for OIDC from the Management Console.

Custom firewall rules are removed during the upgrade process.

During the validation phase of a configuration run, a No such object error may occur for the Notebook and Viewscreen services. This error can be ignored as the services should still correctly start.

If the root site administrator is locked out of the Management Console after failed login attempts, the account does not unlock automatically after the defined lockout time. Someone with administrative SSH access to the instance must unlock the account using the administrative shell. For more information, see "Troubleshooting access to the Management Console."

The mbind: Operation not permitted error in the /var/log/mysql/mysql.err file can be ignored. MySQL 8 does not gracefully handle when the CAP_SYS_NICE capability isn't required, and outputs an error instead of a warning.

After an administrator upgrades from GitHub Enterprise Server 3.7 or 3.8, to 3.9 or 3.10, MySQL may not start back up. For more information, see "Known issues with upgrades to your instance."

After an administrator upgrades from GitHub Enterprise Server 3.7 or 3.8 to 3.9 or 3.10, I/O utilization will increase, and in some cases the instance's performance will be impacted. Reduced performance is due to the database server being upgraded from MySQL 5.7 to MySQL 8.0. For more information, see "Known issues with upgrades to your instance."

In rare circumstances, a small instance with both high availability and GitHub Actions configured may report that MSSQL replication is unhealthy after many upgrades with full upgrade packages. If you encounter this issue, contact GitHub Support.

On an instance in a cluster configuration with high availability configured, ghe-config-apply times out while waiting for hookshot-go to start on replica application nodes.

On an instance hosted in AWS, system time may lose synchronization with Amazon's servers after an administrator reboots the instance.

On an instance with the HTTP X-Forwarded-For header configured for use behind a load balancer, all client IP addresses in the instance's audit log erroneously appear as 127.0.0.1.

In GitHub Enterprise Server 3.10 and later, the requirements for TLS security levels have changed due to an upgrade to containers in the underlying OS. On an instance with GitHub Actions enabled and a custom TLS certificate, users may experience disruptions with workflow runs if the TLS certificate uses weak encryption. Workflow runs will not trigger, and the following error message will appear in system logs for babeld.

CA certificate key too weak

To resolve this issue, confirm that your certificate complies with level 2 of the OpenSSL security specification. For more information, see SSL_CTX_set_security_level in the OpenSSL docs. For more information about reviewing your instance's logs, see "About system logs".

If the error appears in babeld logs because your TLS certificate does not comply with level 2 of the specification, you must create and upload a new certificate with stronger security before you upgrade to GitHub Enterprise Server 3.10 or later. For more information, see "Configuring TLS."

In some situations, large .adoc files stored in a repository do not render properly in the web UI. The raw contents are still available to view as plaintext.

The HAProxy version has been updated in this release. You may encounter elevated error rates when HAProxy is upgraded as part of a hotpatch upgrade to a GitHub Enterprise Server instance. These elevated error rates should resolve within 5 minutes of the hotpatch being applied.

Please note, when performing a hotpatch upgrade to GitHub Enterprise Server version 3.10.4 or higher you will encounter this known issue only if you are hotpatching from GitHub Enterprise Server version 3.10.3 or lower .

The reply.[hostname] subdomain is falsely always displaying as having no SSL and DNS record, when testing the domain settings via management console without subdomain isolation.

Admin stats REST API endpoints may timeout on appliances with many users or repositories. Retrying the request until data is returned is advised.

If a hotpatch upgrade requires the haproxy-frontend service to be restarted, the restart will hang if there are existing long-lived connections, such as browser web sockets or Git operations. No new connections will be accepted for up to 5 minutes. Any existing unfinished connections at this time will be disconnected.

When restoring from a backup snapshot, a large number of mapper_parsing_exception errors may be displayed.

Services may respond with a 503 status due to an out of date haproxy configuration. This can usually be resolved with a ghe-config-apply run.

Users can view the app state of gists, networks, and wikis in the spokesctl info output, enhancing visibility into the status of these elements. Additionally, spokesctl check can diagnose and, in most cases, fix empty repository networks, improving network management.

CRITICAL: On GitHub Enterprise Server instances that use SAML single sign-on (SSO) authentication with specific IdPs utilizing publicly exposed signed federation metadata XML, an attacker could forge a SAML response to provision and/or gain access to a user account with site administrator privileges. GitHub has requested CVE ID CVE-2024-6800 for this vulnerability, which was reported via the GitHub Bug Bounty program.

MEDIUM: An attacker could disclose the issue contents from a private repository using a GitHub App with only contents: read and pull requests: write permissions. This was only exploitable via user access token, and installation access tokens were not impacted. GitHub has requested CVE ID CVE-2024-6337 for this vulnerability, which was reported via the GitHub Bug Bounty program.

Packages have been updated to the latest security versions.

On an instance with GitHub Actions enabled, during a hotpatch upgrade, a race condition could block various upgrade activities.

The ghe-config-apply process made an unnecessary number of connections to Redis.

Instances installed on Google Cloud Platform (GCP) could have their hostname overwritten by GCP when a hotpatch was applied.

The minimum password requirements for Management Console users and the root site administrator required an upper case character when providing a password with a minimum of 8 characters, contradicting the documentation and password hint.

On an instance with subdomain isolation enabled, configuration runs created subdomains for ChatOps services, such as slack.HOSTNAME and teams.HOSTNAME, regardless of whether the service was enabled.

On an instance with GitHub Actions enabled, due to an insufficient wait time, MS SQL and MySQL replication could fail with the error message Failed to start nomad service!.

Some users were unable to delete project views.

Due to a regression introduced in a previous patch, for enterprises that use encrypted SAML assertions, SSO attempts failed with a digest mismatch error if the entire SAML response was signed, rather than just the assertions.

Running go get for a Golang repository with a directory structure that overlaps with GitHub UI routes failed

The github-stream-processor service could get into a state where it would continually fail to process messages with a TRILOGY_CLOSED_CONNECTION error.

A corrupted entry in the Git audit log could cause out of memory errors.

Fixes and improvements for the git core module.

Actions KPI logs are disabled by default to reduce log size.

Audit log events related to audit log streaming are available in the enterprise audit log page, and via audit log streaming.

Custom firewall rules are removed during the upgrade process.

During the validation phase of a configuration run, a No such object error may occur for the Notebook and Viewscreen services. This error can be ignored as the services should still correctly start.

If the root site administrator is locked out of the Management Console after failed login attempts, the account does not unlock automatically after the defined lockout time. Someone with administrative SSH access to the instance must unlock the account using the administrative shell. For more information, see "Troubleshooting access to the Management Console."

If an instance is configured to forward logs to a target server with TLS enabled, certificate authority (CA) bundles that a site administrator uploads using ghe-ssl-ca-certificate-install are not respected, and connections to the server fail.

The mbind: Operation not permitted error in the /var/log/mysql/mysql.err file can be ignored. MySQL 8 does not gracefully handle when the CAP_SYS_NICE capability isn't required, and outputs an error instead of a warning.

After an administrator upgrades from GitHub Enterprise Server 3.7 or 3.8, to 3.9 or 3.10, MySQL may not start back up. For more information, see "Known issues with upgrades to your instance."

After an administrator upgrades from GitHub Enterprise Server 3.7 or 3.8 to 3.9 or 3.10, I/O utilization will increase, and in some cases the instance's performance will be impacted. Reduced performance is due to the database server being upgraded from MySQL 5.7 to MySQL 8.0. For more information, see "Known issues with upgrades to your instance."

In rare circumstances, a small instance with both high availability and GitHub Actions configured may report that MSSQL replication is unhealthy after many upgrades with full upgrade packages. If you encounter this issue, contact GitHub Support.

On an instance in a cluster configuration with high availability configured, ghe-config-apply times out while waiting for hookshot-go to start on replica application nodes.

After an administrator enables maintenance mode from the instance's Management Console UI using Firefox, the administrator is redirected to the Settings page, but maintenance mode is not enabled. To work around this issue, use a different browser.

On an instance hosted in AWS, system time may lose synchronization with Amazon's servers after an administrator reboots the instance.

On an instance with the HTTP X-Forwarded-For header configured for use behind a load balancer, all client IP addresses in the instance's audit log erroneously appear as 127.0.0.1.

In GitHub Enterprise Server 3.10 and later, the requirements for TLS security levels have changed due to an upgrade to containers in the underlying OS. On an instance with GitHub Actions enabled and a custom TLS certificate, users may experience disruptions with workflow runs if the TLS certificate uses weak encryption. Workflow runs will not trigger, and the following error message will appear in system logs for babeld.

CA certificate key too weak

To resolve this issue, confirm that your certificate complies with level 2 of the OpenSSL security specification. For more information, see SSL_CTX_set_security_level in the OpenSSL docs. For more information about reviewing your instance's logs, see "About system logs".

If the error appears in babeld logs because your TLS certificate does not comply with level 2 of the specification, you must create and upload a new certificate with stronger security before you upgrade to GitHub Enterprise Server 3.10 or later. For more information, see "Configuring TLS."

On an instance with GitHub Actions enabled, after an upgrade from GitHub Enterprise Server 3.8 or earlier, an internal exception could prevent successful completion of some operations, like upgrades or the configuration of new replica nodes for high availability. If this issue occurs, administrators may see the following error in /data/user/common/ghe-config.log.

Error occurred while executing servicing step 'Clone datatier login to secondary replica' for component CopyAvailabilityGroupSqlLogins during CopyAvailabilityGroupSqlLogins: Object reference not set to an instance of an object.

To resolve this issue, upgrade to the latest patch release of GitHub Enterprise Server.

In some situations, large .adoc files stored in a repository do not render properly in the web UI. The raw contents are still available to view as plaintext.

On an instance in a cluster configuration, restoration of a backup using ghe-restore will exit prematurely if Redis has not restarted properly.

The HAProxy version has been updated in this release. You may encounter elevated error rates when HAProxy is upgraded as part of a hotpatch upgrade to a GitHub Enterprise Server instance. These elevated error rates should resolve within 5 minutes of the hotpatch being applied.

Please note, when performing a hotpatch upgrade to GitHub Enterprise Server version 3.10.4 or higher you will encounter this known issue only if you are hotpatching from GitHub Enterprise Server version 3.10.3 or lower .

The reply.HOSTNAME subdomain is falsely displayed as having no SSL and DNS record, when testing the domain settings via the Management Console without subdomain isolation.

Admin stats REST API endpoints may timeout on appliances with many users or repositories. Retrying the request until data is returned is advised.

If a hotpatch upgrade requires the haproxy-frontend service to be restarted, the restart will hang if there are existing long-lived connections, such as browser web sockets or Git operations. No new connections will be accepted for up to 5 minutes. Any existing unfinished connections at this time will be disconnected.

When restoring from a backup snapshot, a large number of mapper_parsing_exception errors may be displayed.

Services may respond with a 503 status due to an out of date haproxy configuration. This can usually be resolved with a ghe-config-apply run.

These release notes previously indicated as a known issue that on GitHub Enterprise Server 3.10.16 when log forwarding is enabled, some forwarded log entries may be duplicated. The fix for this problem was already included in GitHub Enterprise Server 3.10.15. [Updated: 2024-09-16]

HIGH: An attacker could cause unbounded resource exhaustion on the instance by sending a large payload to the Git server. To mitigate this issue, GitHub has limited the count of "have" and "want" lines for Git read operations. GitHub has requested CVE ID CVE-2024-5795 for this vulnerability, which was reported via the GitHub Bug Bounty program.

MEDIUM: An improper privilege management vulnerability allowed users to migrate private repositories without having appropriate scopes defined on the related personal access token. GitHub has requested CVE ID CVE-2024-5566 for this vulnerability, which was reported via the GitHub Bug Bounty program.

MEDIUM: An attacker could have unauthorized access in a public repository using a suspended GitHub App via a scoped user access token. This was only exploitable in public repositories while private repositories were not impacted. GitHub has requested CVE ID CVE-2024-5816 for this vulnerability, which was reported via the GitHub Bug Bounty program.

MEDIUM: An attacker could execute a Cross Site Request Forgery (CSRF) attack to perform write operations on a victim-owned repository in GitHub Enterprise Server by exploiting incorrect request types. A mitigating factor is that the attacker has to be a trusted user and the victim has to visit a tag in the attacker's fork of their own repository. GitHub has requested CVE ID CVE-2024-5815 for this vulnerability, which was reported via the GitHub Bug Bounty program.

MEDIUM: An attacker could disclose the name of a private repository on the GitHub Enterprise Server appliance when the private repository has a deploy key associated to it. GitHub has requested CVE ID CVE-2024-6395 for this vulnerability, which was reported via the GitHub Bug Bounty program.

LOW: Instance administrators could see fine-grained personal access tokens in plaintext in the babeld and gitauth logs.

LOW: An attacker with read access to a project could use the REST API to view a list of all members in an organization, including members who had made their membership private. This vulnerability was reported via the GitHub Bug Bounty program.

LOW: An attacker could include MathJax syntax in Markdown to bypass GitHubs normal restrictions on CSS properties in Markdown. This vulnerability was reported via the GitHub Bug Bounty program.

MEDIUM: An attacker could disclose sensitive information from a private repository exploiting organization ruleset features. This attack required an organization member to explicitly change the visibility of a dependent repository from private to public. GitHub has requested CVE ID CVE-2024-6336 for this vulnerability, which was reported via the GitHub Bug Bounty program.

MEDIUM: An attacker could have unauthorized read access to issue content inside an internal repository via GitHub projects. This attack required attacker access to the corresponding project board. GitHub has requested CVE ID CVE-2024-5817 for this vulnerability, which was reported via the GitHub Bug Bounty program.

Firewall port 9199, which linked to a static maintenance page used when enabling maintenance mode with an IP exception list, was opened unnecessarily.

Packages have been updated to the latest security versions.

When an instance hosted on Azure was upgraded with a hotpatch, the upgrade failed with an rsync error.

On an instance with GitHub Actions enabled, remote blob storage could fill up with large amounts of data because cleanup jobs were skipped on old hosts.

In some cases, commands run in an administrative SSH shell were not written to the audit log.

When an administrator submitted support data to GitHub Support, spokesd keys were incorrectly sanitized.

When log forwarding was enabled, some specific service logs, including babeld, gitauth, unicorn, and resqued, were duplicated.

During the initial boot of an instance, a data disk attached as /dev/sdb may not have been recognized as an available disk.

In some cases, the HAProxy kill_timeout setting caused service outages during upgrades or large transactions.

The ssh-audit-log.sh script did not effectively log SSH commands, and the ghe-sanitize-log.psed script inadequately sanitized password-related logs.

The default MSSQL timeout of 8 seconds sometimes caused issues during administrator activities. The default timeout has been increased to 30 seconds.

For an instance running on Microsoft Azure, the user disk service failed to start because the attached volume could not be found.

Establishing a new GitHub Connect connection could fail with a 500 error.

When using ghe-migrator to migrate a repository, the links for pull requests merge commits were not imported.

In some cases, reading data from repositories with a large number of objects would result in timeout or error.

When a user used the REST API endpoints that returned secret scanning alerts at the repository or organization level with non-cursor-based pagination (for example, without before or after query parameters), the REST API endpoints for secret scanning returned incorrect Link headers.

On instances with SAML authentication configured, users were unable to sign out and became stuck in an infinite SAML SSO loop.

Deleting a branch that was targeted by many pull requests could result in delayed job processing and increased system memory usage.

On an instance that restricts emails to verified domains, secret scanning emails would sometimes be sent to an unverified domain.

In some cases, on the "Files" tab of a pull request, a comment on the first line did not render.

Some organizations were not recognized as part of an instance's enterprise account.

Some users would encounter an error when navigating to their personal security settings page at https://HOSTNAME/settings/security.

On the "Code scanning" page of a repository, the branch filter did not correctly display all branches.

Users viewing the alerts index page experienced inconsistencies in rendering the closed alert state.

Organizations named "C" were incorrectly routed to the GitHub Enterprise Server contact page instead of their organization page.

When servers responded with unsupported characters, webhook deliveries were not displayed in the UI.

Chat integrations required frequent reauthentication, as a result of new app installations overwriting previous ones.

On an instance in a cluster configuration, the ghe-spokesctl ssh command did not select the correct Nomad container when running a command within a git repository.

On an instance with a GitHub Advanced Security license, disabling and re-enabling GitHub Advanced Security for an organization resulted in redundant scans of some repositories.

The timeout for requests made to the REST API endpoints for secret scanning has been extended.

When a user changes a repository's visibility to public, the user is now warned that previous Actions history and logs will become public as well.

When using the ghe-webhook-logs utility, webhook delivery logs can be filtered by event and action. Users can use ghe-webhook-logs --event issues to filter by event, or ghe-webhook-logs --event issues.opened to filter by event and action.

Custom firewall rules are removed during the upgrade process.

During the validation phase of a configuration run, a No such object error may occur for the Notebook and Viewscreen services. This error can be ignored as the services should still correctly start.

If the root site administrator is locked out of the Management Console after failed login attempts, the account does not unlock automatically after the defined lockout time. Someone with administrative SSH access to the instance must unlock the account using the administrative shell. For more information, see "Troubleshooting access to the Management Console."

If an instance is configured to forward logs to a target server with TLS enabled, certificate authority (CA) bundles that a site administrator uploads using ghe-ssl-ca-certificate-install are not respected, and connections to the server fail.

The mbind: Operation not permitted error in the /var/log/mysql/mysql.err file can be ignored. MySQL 8 does not gracefully handle when the CAP_SYS_NICE capability isn't required, and outputs an error instead of a warning.

After an administrator upgrades from GitHub Enterprise Server 3.7 or 3.8, to 3.9 or 3.10, MySQL may not start back up. For more information, see "Known issues with upgrades to your instance."

After an administrator upgrades from GitHub Enterprise Server 3.7 or 3.8 to 3.9 or 3.10, I/O utilization will increase, and in some cases the instance's performance will be impacted. Reduced performance is due to the database server being upgraded from MySQL 5.7 to MySQL 8.0. For more information, see "Known issues with upgrades to your instance."

In rare circumstances, a small instance with both high availability and GitHub Actions configured may report that MSSQL replication is unhealthy after many upgrades with full upgrade packages. If you encounter this issue, contact GitHub Support.

On an instance in a cluster configuration with high availability configured, ghe-config-apply times out while waiting for hookshot-go to start on replica application nodes.

After an administrator enables maintenance mode from the instance's Management Console UI using Firefox, the administrator is redirected to the Settings page, but maintenance mode is not enabled. To work around this issue, use a different browser.

On an instance hosted in AWS, system time may lose synchronization with Amazon's servers after an administrator reboots the instance.

On an instance with the HTTP X-Forwarded-For header configured for use behind a load balancer, all client IP addresses in the instance's audit log erroneously appear as 127.0.0.1.

In GitHub Enterprise Server 3.10 and later, the requirements for TLS security levels have changed due to an upgrade to containers in the underlying OS. On an instance with GitHub Actions enabled and a custom TLS certificate, users may experience disruptions with workflow runs if the TLS certificate uses weak encryption. Workflow runs will not trigger, and the following error message will appear in system logs for babeld.

CA certificate key too weak

To resolve this issue, confirm that your certificate complies with level 2 of the OpenSSL security specification. For more information, see SSL_CTX_set_security_level in the OpenSSL docs. For more information about reviewing your instance's logs, see "About system logs".

If the error appears in babeld logs because your TLS certificate does not comply with level 2 of the specification, you must create and upload a new certificate with stronger security before you upgrade to GitHub Enterprise Server 3.10 or later. For more information, see "Configuring TLS."

On an instance with GitHub Actions enabled, after an upgrade from GitHub Enterprise Server 3.8 or earlier, an internal exception could prevent successful completion of some operations, like upgrades or the configuration of new replica nodes for high availability. If this issue occurs, administrators may see the following error in /data/user/common/ghe-config.log.

Error occurred while executing servicing step 'Clone datatier login to secondary replica' for component CopyAvailabilityGroupSqlLogins during CopyAvailabilityGroupSqlLogins: Object reference not set to an instance of an object.

To resolve this issue, upgrade to the latest patch release of GitHub Enterprise Server.

In some situations, large .adoc files stored in a repository do not render properly in the web UI. The raw contents are still available to view as plaintext.

On an instance in a cluster configuration, restoration of a backup using ghe-restore will exit prematurely if Redis has not restarted properly.

The HAProxy version has been updated in this release. You may encounter elevated error rates when HAProxy is upgraded as part of a hotpatch upgrade to a GitHub Enterprise Server instance. These elevated error rates should resolve within 5 minutes of the hotpatch being applied.

Please note, when performing a hotpatch upgrade to GitHub Enterprise Server version 3.10.4 or higher you will encounter this known issue only if you are hotpatching from GitHub Enterprise Server version 3.10.3 or lower .

The reply.[hostname] subdomain is falsely always displaying as having no ssl and dns record, when testing the domain settings via management console without subdomain isolation.

Admin stats REST API endpoints may timeout on appliances with many users or repositories. Retrying the request until data is returned is advised.

If a hotpatch upgrade requires the haproxy-frontend service to be restarted, the restart will hang if there are existing long-lived connections, such as browser web sockets or Git operations. No new connections will be accepted for up to 5 minutes. Any existing unfinished connections at this time will be disconnected.

HIGH: An attacker with the site administrator role could gain arbitrary code execution capability on the GitHub Enterprise Server appliance when configuring audit log streaming. GitHub has requested CVE ID CVE-2024-5746 for this vulnerability, which was reported via the GitHub Bug Bounty program.

Packages have been updated to the latest security versions.

On an instance with GitHub Actions and External MySQL enabled, a validation step in the config apply could fail.

Custom firewall rules are removed during the upgrade process.

During the validation phase of a configuration run, a No such object error may occur for the Notebook and Viewscreen services. This error can be ignored as the services should still correctly start.

If the root site administrator is locked out of the Management Console after failed login attempts, the account does not unlock automatically after the defined lockout time. Someone with administrative SSH access to the instance must unlock the account using the administrative shell. For more information, see "Troubleshooting access to the Management Console."

If an instance is configured to forward logs to a target server with TLS enabled, certificate authority (CA) bundles that a site administrator uploads using ghe-ssl-ca-certificate-install are not respected, and connections to the server fail.

The mbind: Operation not permitted error in the /var/log/mysql/mysql.err file can be ignored. MySQL 8 does not gracefully handle when the CAP_SYS_NICE capability isn't required, and outputs an error instead of a warning.

After an administrator upgrades from GitHub Enterprise Server 3.7 or 3.8, to 3.9 or 3.10, MySQL may not start back up. For more information, see "Known issues with upgrades to your instance."

After an administrator upgrades from GitHub Enterprise Server 3.7 or 3.8 to 3.9 or 3.10, I/O utilization will increase, and in some cases the instance's performance will be impacted. Reduced performance is due to the database server being upgraded from MySQL 5.7 to MySQL 8.0. For more information, see "Known issues with upgrades to your instance."

In rare circumstances, a small instance with both high availability and GitHub Actions configured may report that MSSQL replication is unhealthy after many upgrades with full upgrade packages. If you encounter this issue, contact GitHub Support.

On an instance in a cluster configuration with high availability configured, ghe-config-apply times out while waiting for hookshot-go to start on replica application nodes.

After an administrator enables maintenance mode from the instance's Management Console UI using Firefox, the administrator is redirected to the Settings page, but maintenance mode is not enabled. To work around this issue, use a different browser.

On an instance hosted in AWS, system time may lose synchronization with Amazon's servers after an administrator reboots the instance.

On an instance with the HTTP X-Forwarded-For header configured for use behind a load balancer, all client IP addresses in the instance's audit log erroneously appear as 127.0.0.1.

In GitHub Enterprise Server 3.10 and later, the requirements for TLS security levels have changed due to an upgrade to containers in the underlying OS. On an instance with GitHub Actions enabled and a custom TLS certificate, users may experience disruptions with workflow runs if the TLS certificate uses weak encryption. Workflow runs will not trigger, and the following error message will appear in system logs for babeld.

CA certificate key too weak

To resolve this issue, confirm that your certificate complies with level 2 of the OpenSSL security specification. For more information, see SSL_CTX_set_security_level in the OpenSSL docs. For more information about reviewing your instance's logs, see "About system logs".

If the error appears in babeld logs because your TLS certificate does not comply with level 2 of the specification, you must create and upload a new certificate with stronger security before you upgrade to GitHub Enterprise Server 3.10 or later. For more information, see "Configuring TLS."

On an instance with GitHub Actions enabled, after an upgrade from GitHub Enterprise Server 3.8 or earlier, an internal exception could prevent successful completion of some operations, like upgrades or the configuration of new replica nodes for high availability. If this issue occurs, administrators may see the following error in /data/user/common/ghe-config.log.

Error occurred while executing servicing step 'Clone datatier login to secondary replica' for component CopyAvailabilityGroupSqlLogins during CopyAvailabilityGroupSqlLogins: Object reference not set to an instance of an object.

To resolve this issue, upgrade to the latest patch release of GitHub Enterprise Server.

In some situations, large .adoc files stored in a repository do not render properly in the web UI. The raw contents are still available to view as plaintext.

On an instance in a cluster configuration, restoration of a backup using ghe-restore will exit prematurely if Redis has not restarted properly.

The HAProxy version has been updated in this release. You may encounter elevated error rates when HAProxy is upgraded as part of a hotpatch upgrade to a GitHub Enterprise Server instance. These elevated error rates should resolve within 5 minutes of the hotpatch being applied.

Please note, when performing a hotpatch upgrade to GitHub Enterprise Server version 3.10.4 or higher you will encounter this known issue only if you are hotpatching from GitHub Enterprise Server version 3.10.3 or lower .

When enabling log forwarding, specific services logs (babeld and some more) are duplicated.

The reply.[hostname] subdomain is falsely always displaying as having no SSL and DNS record, when testing the domain settings via management console without subdomain isolation.

When log forwarding is enabled, some forwarded log entries may be duplicated.

Admin stats REST API endpoints may timeout on appliances with many users or repositories. Retrying the request until data is returned is advised.

If a hotpatch upgrade requires the haproxy-frontend service to be restarted, the restart will hang if there are existing long-lived connections, such as browser web sockets or Git operations. No new connections will be accepted for up to 5 minutes. Any existing unfinished connections at this time will be disconnected.

CRITICAL: On instances that use SAML single sign-on (SSO) authentication with the optional encrypted assertions feature, an attacker could forge a SAML response to provision and/or gain access to a user with administrator privileges.

Please note that encrypted assertions are not enabled by default. Instances not utilizing SAML SSO or utilizing SAML SSO authentication without encrypted assertions are not impacted. Exploitation of this vulnerability would allow unauthorized access to the instance without requiring prior authentication. GitHub has requested CVE ID CVE-2024-4985 for this vulnerability, which was reported via the GitHub Bug Bounty program.

For more information, see "Configuring SAML single sign-on for your enterprise" and "Enabling encrypted assertions."

Custom firewall rules are removed during the upgrade process.

During the validation phase of a configuration run, a No such object error may occur for the Notebook and Viewscreen services. This error can be ignored as the services should still correctly start.

If the root site administrator is locked out of the Management Console after failed login attempts, the account does not unlock automatically after the defined lockout time. Someone with administrative SSH access to the instance must unlock the account using the administrative shell. For more information, see "Troubleshooting access to the Management Console."

If an instance is configured to forward logs to a target server with TLS enabled, certificate authority (CA) bundles that a site administrator uploads using ghe-ssl-ca-certificate-install are not respected, and connections to the server fail.

The mbind: Operation not permitted error in the /var/log/mysql/mysql.err file can be ignored. MySQL 8 does not gracefully handle when the CAP_SYS_NICE capability isn't required, and outputs an error instead of a warning.

After an administrator upgrades from GitHub Enterprise Server 3.7 or 3.8, to 3.9 or 3.10, MySQL may not start back up. For more information, see "Known issues with upgrades to your instance."

After an administrator upgrades from GitHub Enterprise Server 3.7 or 3.8 to 3.9 or 3.10, I/O utilization will increase, and in some cases the instance's performance will be impacted. Reduced performance is due to the database server being upgraded from MySQL 5.7 to MySQL 8.0. For more information, see "Known issues with upgrades to your instance."

In rare circumstances, a small instance with both high availability and GitHub Actions configured may report that MSSQL replication is unhealthy after many upgrades with full upgrade packages. If you encounter this issue, contact GitHub Support.

On an instance in a cluster configuration with high availability configured, ghe-config-apply times out while waiting for hookshot-go to start on replica application nodes.

After an administrator enables maintenance mode from the instance's Management Console UI using Firefox, the administrator is redirected to the Settings page, but maintenance mode is not enabled. To work around this issue, use a different browser.

On an instance hosted in AWS, system time may lose synchronization with Amazon's servers after an administrator reboots the instance.

On an instance with the HTTP X-Forwarded-For header configured for use behind a load balancer, all client IP addresses in the instance's audit log erroneously appear as 127.0.0.1.

In GitHub Enterprise Server 3.10 and later, the requirements for TLS security levels have changed due to an upgrade to containers in the underlying OS. On an instance with GitHub Actions enabled and a custom TLS certificate, users may experience disruptions with workflow runs if the TLS certificate uses weak encryption. Workflow runs will not trigger, and the following error message will appear in system logs for babeld.

CA certificate key too weak

To resolve this issue, confirm that your certificate complies with level 2 of the OpenSSL security specification. For more information, see SSL_CTX_set_security_level in the OpenSSL docs. For more information about reviewing your instance's logs, see "About system logs".

If the error appears in babeld logs because your TLS certificate does not comply with level 2 of the specification, you must create and upload a new certificate with stronger security before you upgrade to GitHub Enterprise Server 3.10 or later. For more information, see "Configuring TLS."

On an instance with GitHub Actions enabled, after an upgrade from GitHub Enterprise Server 3.8 or earlier, an internal exception could prevent successful completion of some operations, like upgrades or the configuration of new replica nodes for high availability. If this issue occurs, administrators may see the following error in /data/user/common/ghe-config.log.

Error occurred while executing servicing step 'Clone datatier login to secondary replica' for component CopyAvailabilityGroupSqlLogins during CopyAvailabilityGroupSqlLogins: Object reference not set to an instance of an object.

To resolve this issue, upgrade to the latest patch release of GitHub Enterprise Server.

In some situations, large .adoc files stored in a repository do not render properly in the web UI. The raw contents are still available to view as plaintext.

On an instance in a cluster configuration, restoration of a backup using ghe-restore will exit prematurely if Redis has not restarted properly.

The HAProxy version has been updated in this release. You may encounter elevated error rates when HAProxy is upgraded as part of a hotpatch upgrade to a GitHub Enterprise Server instance. These elevated error rates should resolve within 5 minutes of the hotpatch being applied.

Please note, when performing a hotpatch upgrade to GitHub Enterprise Server version 3.10.4 or higher you will encounter this known issue only if you are hotpatching from GitHub Enterprise Server version 3.10.3 or lower .

If a hotpatch upgrade requires the haproxy-frontend service to be restarted, the restart will hang if there are existing long-lived connections, such as browser web sockets or Git operations. No new connections will be accepted for up to 5 minutes. Any existing unfinished connections at this time will be disconnected. [Updated: 2024-06-17]

As a result of a security vulnerability, the editor role for a Management Console user has been deprecated in the Manage GitHub Enterprise Server API.

Packages have been updated to the latest security versions.

Running ghe-repl-node -d did not validate value length in order to prevent values longer than 20 characters.

For an instance in a cluster configuration, during the migration phase of a configuration run, the process of copying configuration updates to all nodes would fail.

External collaborators with read-only access were able to run workflows on their pull requests from private forks without approval.

On an instance with a GitHub Advanced Security license, custom pattern matches were incorrectly filtered during post-scan filtering.

To aid in understanding the CPU/memory utilization of secret scanning processes, the binary names of nomad workers were updated to differentiate between the different types of secret scanning jobs.

A more specific error message is shown when the ghe-repl-node command is run on an instance not configured for high availability.

Custom firewall rules are removed during the upgrade process.

During the validation phase of a configuration run, a No such object error may occur for the Notebook and Viewscreen services. This error can be ignored as the services should still correctly start.

If the root site administrator is locked out of the Management Console after failed login attempts, the account does not unlock automatically after the defined lockout time. Someone with administrative SSH access to the instance must unlock the account using the administrative shell. For more information, see "Troubleshooting access to the Management Console."

If an instance is configured to forward logs to a target server with TLS enabled, certificate authority (CA) bundles that a site administrator uploads using ghe-ssl-ca-certificate-install are not respected, and connections to the server fail.

The mbind: Operation not permitted error in the /var/log/mysql/mysql.err file can be ignored. MySQL 8 does not gracefully handle when the CAP_SYS_NICE capability isn't required, and outputs an error instead of a warning.

After an administrator upgrades from GitHub Enterprise Server 3.7 or 3.8, to 3.9 or 3.10, MySQL may not start back up. For more information, see "Known issues with upgrades to your instance."

After an administrator upgrades from GitHub Enterprise Server 3.7 or 3.8 to 3.9 or 3.10, I/O utilization will increase, and in some cases the instance's performance will be impacted. Reduced performance is due to the database server being upgraded from MySQL 5.7 to MySQL 8.0. For more information, see "Known issues with upgrades to your instance."

In rare circumstances, a small instance with both high availability and GitHub Actions configured may report that MSSQL replication is unhealthy after many upgrades with full upgrade packages. If you encounter this issue, contact GitHub Support.

On an instance in a cluster configuration with high availability configured, ghe-config-apply times out while waiting for hookshot-go to start on replica application nodes.

After an administrator enables maintenance mode from the instance's Management Console UI using Firefox, the administrator is redirected to the Settings page, but maintenance mode is not enabled. To work around this issue, use a different browser.

On an instance hosted in AWS, system time may lose synchronization with Amazon's servers after an administrator reboots the instance.

On an instance with the HTTP X-Forwarded-For header configured for use behind a load balancer, all client IP addresses in the instance's audit log erroneously appear as 127.0.0.1.

In GitHub Enterprise Server 3.10 and later, the requirements for TLS security levels have changed due to an upgrade to containers in the underlying OS. On an instance with GitHub Actions enabled and a custom TLS certificate, users may experience disruptions with workflow runs if the TLS certificate uses weak encryption. Workflow runs will not trigger, and the following error message will appear in system logs for babeld.

CA certificate key too weak

To resolve this issue, confirm that your certificate complies with level 2 of the OpenSSL security specification. For more information, see SSL_CTX_set_security_level in the OpenSSL docs. For more information about reviewing your instance's logs, see "About system logs".

If the error appears in babeld logs because your TLS certificate does not comply with level 2 of the specification, you must create and upload a new certificate with stronger security before you upgrade to GitHub Enterprise Server 3.10 or later. For more information, see "Configuring TLS."

On an instance with GitHub Actions enabled, after an upgrade from GitHub Enterprise Server 3.8 or earlier, an internal exception could prevent successful completion of some operations, like upgrades or the configuration of new replica nodes for high availability. If this issue occurs, administrators may see the following error in /data/user/common/ghe-config.log.

Error occurred while executing servicing step 'Clone datatier login to secondary replica' for component CopyAvailabilityGroupSqlLogins during CopyAvailabilityGroupSqlLogins: Object reference not set to an instance of an object.

To resolve this issue, upgrade to the latest patch release of GitHub Enterprise Server.

In some situations, large .adoc files stored in a repository do not render properly in the web UI. The raw contents are still available to view as plaintext.

On an instance in a cluster configuration, restoration of a backup using ghe-restore will exit prematurely if Redis has not restarted properly.

The HAProxy version has been updated in this release. You may encounter elevated error rates when HAProxy is upgraded as part of a hotpatch upgrade to a GitHub Enterprise Server instance. These elevated error rates should resolve within 5 minutes of the hotpatch being applied.

Please note, when performing a hotpatch upgrade to GitHub Enterprise Server version 3.10.4 or higher you will encounter this known issue only if you are hotpatching from GitHub Enterprise Server version 3.10.3 or lower .

If a hotpatch upgrade requires the haproxy-frontend service to be restarted, the restart will hang if there are existing long-lived connections, such as browser web sockets or Git operations. No new connections will be accepted for up to 5 minutes. Any existing unfinished connections at this time will be disconnected. [Updated: 2024-06-17]

HIGH: An attacker with the editor role in the Management Console could gain administrative SSH access to the appliance by command injection when configuring the chat integration. GitHub has requested CVE ID CVE-2024-3646 for this vulnerability, which was reported via the GitHub Bug Bounty program. The editor role has been deprecated. For more information, see the "Changes" section of these release notes.

HIGH: An attacker with an editor role in the Management Console could gain SSH access to the instance by command injection when configuring Artifact & Logs and Migrations Storage. GitHub has requested CVE ID CVE-2024-3684 for this vulnerability, which was reported via the GitHub Bug Bounty program.

MEDIUM: An attacker could maintain admin access to a detached repository in a race condition by making a GraphQL mutation to alter repository permissions while the repository is detached. GitHub has requested CVE ID CVE-2024-2440 for this vulnerability, which was reported via the GitHub Bug Bounty program.

A GraphQL endpoint was disabled as part of a previous security fix, causing errors with the "Auto-add to project" workflow and with issue creation from within a project. To resolve these errors, a security patch has been applied and the affected GraphQL endpoint has been re-enabled.

Packages have been updated to the latest security versions.

When configuring audit log streaming to Datadog or Splunk on an instance with custom CA certificates, the connection failed with the error There was an error trying to connect.

Disk usage, utilization, and latency for data devices could render incorrectly in Grafana.

On an instance in a cluster configuration, former primary nodes were able to access the newly promoted nodes after failover. The ghe-cluster-failover command has been updated to block access from the old cluster, and four new command-line utilities have been introduced to manually block IP addresses: ghe-cluster-block-ips, ghe-cluster-block-ip, ghe-cluster-unblock-ips, and ghe-cluster-unblock-ip. For more information, see "Command-line utilities." [Updated: 2024-05-01]

The ghe-update-check command did not clean up .tmp files in /var/lib/ghe-updates/, which could lead to full disk issues.

On an instance that failed a configuration run, when attempting to repeat the restore step of a backup, the audit log restore step returned error lines even though audit logs were being fully restored.

In some cases, Treelights timeouts caused pull requests to return a 500 error.

The web UI presented inapplicable fine-grained permissions for assignment to custom repository roles. The permissions were also displayed as implicitly included in certain base roles.

On an instance with a GitHub Advanced Security license, some searches for secret scanning alerts resulted in a 500 error.

The profile settings for organizations displayed a warning about profile images that does not apply to organizations on a GitHub Enterprise Server instance.

Administrators could get a 500 error when trying to access the "File storage" section of the site admin dashboard.

Setting a maintenance message failed if the message contained a multibyte character.

On an instance with a GitHub Advanced Security license, metrics for custom patterns alerts incorrectly included tokens in ignored locations.

On an instance with code scanning enabled, on the tool status page for code scanning, outdated upload errors were still displayed after a successful upload.

On an instance where user avatars had been deleted directly from the database, an identicon avatar was not correctly displayed for affected users, and administrators may have observed a relatively high number of application exceptions.

On an instance hosted on Azure, administrators can set and reset SSH keys and passwords via the Azure Agent.

As a result of a security vulnerability, the editor role for a Management Console user has been deprecated. For details, see the "Security fixes" section of these release notes. Existing users with the editor role will be unable to log in to the Management Console, and should contact their site administrator requesting that access be reinstated by updating the user to the operator role if appropriate.

Administrators can improve the performance of "Create a new repository" and "Create a new fork" pages by running this command: ghe-config app.github.create-repo-perf true && ghe-config-apply.

Custom firewall rules are removed during the upgrade process.

During the validation phase of a configuration run, a No such object error may occur for the Notebook and Viewscreen services. This error can be ignored as the services should still correctly start.

If the root site administrator is locked out of the Management Console after failed login attempts, the account does not unlock automatically after the defined lockout time. Someone with administrative SSH access to the instance must unlock the account using the administrative shell. For more information, see "Troubleshooting access to the Management Console."

If an instance is configured to forward logs to a target server with TLS enabled, certificate authority (CA) bundles that a site administrator uploads using ghe-ssl-ca-certificate-install are not respected, and connections to the server fail.

The mbind: Operation not permitted error in the /var/log/mysql/mysql.err file can be ignored. MySQL 8 does not gracefully handle when the CAP_SYS_NICE capability isn't required, and outputs an error instead of a warning.

After an administrator upgrades from GitHub Enterprise Server 3.7 or 3.8, to 3.9 or 3.10, MySQL may not start back up. For more information, see "Known issues with upgrades to your instance."

After an administrator upgrades from GitHub Enterprise Server 3.7 or 3.8 to 3.9 or 3.10, I/O utilization will increase, and in some cases the instance's performance will be impacted. Reduced performance is due to the database server being upgraded from MySQL 5.7 to MySQL 8.0. For more information, see "Known issues with upgrades to your instance."

In rare circumstances, a small instance with both high availability and GitHub Actions configured may report that MSSQL replication is unhealthy after many upgrades with full upgrade packages. If you encounter this issue, contact GitHub Support.

On an instance in a cluster configuration with high availability configured, ghe-config-apply times out while waiting for hookshot-go to start on replica application nodes.

After an administrator enables maintenance mode from the instance's Management Console UI using Firefox, the administrator is redirected to the Settings page, but maintenance mode is not enabled. To work around this issue, use a different browser.

On an instance hosted in AWS, system time may lose synchronization with Amazon's servers after an administrator reboots the instance.

On an instance with the HTTP X-Forwarded-For header configured for use behind a load balancer, all client IP addresses in the instance's audit log erroneously appear as 127.0.0.1.

In GitHub Enterprise Server 3.10 and later, the requirements for TLS security levels have changed due to an upgrade to containers in the underlying OS. On an instance with GitHub Actions enabled and a custom TLS certificate, users may experience disruptions with workflow runs if the TLS certificate uses weak encryption. Workflow runs will not trigger, and the following error message will appear in system logs for babeld.

CA certificate key too weak

To resolve this issue, confirm that your certificate complies with level 2 of the OpenSSL security specification. For more information, see SSL_CTX_set_security_level in the OpenSSL docs. For more information about reviewing your instance's logs, see "About system logs".

If the error appears in babeld logs because your TLS certificate does not comply with level 2 of the specification, you must create and upload a new certificate with stronger security before you upgrade to GitHub Enterprise Server 3.10 or later. For more information, see "Configuring TLS."

On an instance with GitHub Actions enabled, after an upgrade from GitHub Enterprise Server 3.8 or earlier, an internal exception could prevent successful completion of some operations, like upgrades or the configuration of new replica nodes for high availability. If this issue occurs, administrators may see the following error in /data/user/common/ghe-config.log.

Error occurred while executing servicing step 'Clone datatier login to secondary replica' for component CopyAvailabilityGroupSqlLogins during CopyAvailabilityGroupSqlLogins: Object reference not set to an instance of an object.

To resolve this issue, upgrade to the latest patch release of GitHub Enterprise Server.

In some situations, large .adoc files stored in a repository do not render properly in the web UI. The raw contents are still available to view as plaintext.

On an instance in a cluster configuration, restoration of a backup using ghe-restore will exit prematurely if Redis has not restarted properly.

The HAProxy version has been updated in this release. You may encounter elevated error rates when HAProxy is upgraded as part of a hotpatch upgrade to a GitHub Enterprise Server instance. These elevated error rates should resolve within 5 minutes of the hotpatch being applied.

Please note, when performing a hotpatch upgrade to GitHub Enterprise Server version 3.10.4 or higher you will encounter this known issue only if you are hotpatching from GitHub Enterprise Server version 3.10.3 or lower .

If a hotpatch upgrade requires the haproxy-frontend service to be restarted, the restart will hang if there are existing long-lived connections, such as browser web sockets or Git operations. No new connections will be accepted for up to 5 minutes. Any existing unfinished connections at this time will be disconnected. [Updated: 2024-06-17]

HIGH: An attacker with an Administrator role in GitHub Enterprise Server could gain SSH root access via remote code execution. GitHub has requested CVE ID CVE-2024-2469 for this vulnerability, which was reported via the GitHub Bug Bounty program.

HIGH: An attacker with an editor role in the Management Console could gain SSH access to the instance by command injection when configuring GeoJSON settings. GitHub has requested CVE ID CVE-2024-2443 for this vulnerability, which was reported via the GitHub Bug Bounty program.

Packages have been updated to the latest security versions.

In some cases, storage initialization on a new instance launch could cause EBS-backed data volumes to not be detected correctly.

Redundant messages caused an increase in the volume of events logged in /var/log/syslog.

Administrators could initiate an SSH audit that unknowingly unverified all SSH keys.

Attributes used to debug LDAP issues were not included in system logs.

In some cases, the codeload service could panic during shutdown and not terminate gracefully.

On an instance in a cluster configuration with high availability enabled, the ghe-spokesctl command failed when run on a replica node.

If an administrator lost SSH access to an instance, authentication from the hypervisor console using the password for the root site administrator would fail.

On an instance with GitHub Actions enabled, GitHub Actions workflows that deployed GitHub Pages sites failed with the following error: Error: Deployment failed, try again later.

Organizations using projects (classic) returned an error log about a soon-to-be deprecated MySQL feature when viewing a project.

On an instance in a cluster configuration, Jupyter notebooks did not render correctly.

On an instance in a cluster configuration with many nodes, requests to the REST API for managing GitHub Enterprise Server would exceed the instance's HTTP timeouts.

On an instance with a GitHub Advanced Security license, in some cases, when a user deleted a custom pattern for secret scanning, GitHub Enterprise Server failed to close or delete the pattern's alerts.

When an administrator set a policy to require two-factor authentication (2FA) for an enterprise, a message incorrectly indicated that users without 2FA enabled on their account would be removed from the enterprise. These users will be removed from repositories and organizations in the enterprise, but not from the enterprise itself.

Some API endpoints for projects did not properly filter target repositories based on the users access.

On an instance with a GitHub Advanced Security license, viewing a secret scanning alert as a user without the security manager role would return a 500 error if the alert was generated from a Git tag instead of a normal commit.

When using GitHub Enterprise Importer to import repositories, ghost users in archive metadata files would cause an error when generating a list of migration conflicts using ghe-migrator conflicts.

After an administrator ran ghe-saml-mapping-csv, the output did not include the corresponding SQL query.

During a configuration run prompted by the delayed restart of the notebooks service, a container validation warning appeared in system logs.

On an instance in a cluster configuration, rebuilds of GitHub Pages sites failed if no replicas of the GitHub Pages data were available (for example, on a newly restored cluster).

In some cases, manual repository maintenance using ghe-spokesctl would fail with the following error: panic: runtime error: invalid memory address or nil pointer dereference.

On an instance with code scanning enabled, upgrades to GitHub Enterprise Server version 3.9 or 3.10 could be slow if a large number of code scanning analyses were present on the instance.

On an instance with a GitHub Advanced Security license, the speed of migration for code scanning analyses is increased during an upgrade from GitHub Enterprise Server 3.10 or earlier.

Gists can be deleted using the Purge Gist button on the Deleted Gists page in Staff Tools.

People deploying a GitHub Enterprise Server instance in AWS can now deploy in an environment that uses Instance Metadata Service Version 2 (IMDSv2).

On an instance in a cluster configuration, MySQL replica nodes can be configured to skip database seeding. For more information, see "Deferring database seeding."

The payload for the push webhook event is now limited to 2,048 commits. If there are more than 2,048 commits in a push, the payload for the push webhook will not contain serialized diff information for each commit. If you need to fetch commit information, you can use the Commits endpoints of the REST API. For more information, see "Webhook events and payloads" and "REST API endpoints for commits."

Custom firewall rules are removed during the upgrade process.

During the validation phase of a configuration run, a No such object error may occur for the Notebook and Viewscreen services. This error can be ignored as the services should still correctly start.

If the root site administrator is locked out of the Management Console after failed login attempts, the account does not unlock automatically after the defined lockout time. Someone with administrative SSH access to the instance must unlock the account using the administrative shell. For more information, see "Troubleshooting access to the Management Console."

If an instance is configured to forward logs to a target server with TLS enabled, certificate authority (CA) bundles that a site administrator uploads using ghe-ssl-ca-certificate-install are not respected, and connections to the server fail.

The mbind: Operation not permitted error in the /var/log/mysql/mysql.err file can be ignored. MySQL 8 does not gracefully handle when the CAP_SYS_NICE capability isn't required, and outputs an error instead of a warning.

After an administrator upgrades from GitHub Enterprise Server 3.7 or 3.8, to 3.9 or 3.10, MySQL may not start back up. For more information, see "Known issues with upgrades to your instance."

After an administrator upgrades from GitHub Enterprise Server 3.7 or 3.8 to 3.9 or 3.10, I/O utilization will increase, and in some cases the instance's performance will be impacted. Reduced performance is due to the database server being upgraded from MySQL 5.7 to MySQL 8.0. For more information, see "Known issues with upgrades to your instance."

In rare circumstances, a small instance with both high availability and GitHub Actions configured may report that MSSQL replication is unhealthy after many upgrades with full upgrade packages. If you encounter this issue, contact GitHub Support.

On an instance in a cluster configuration with high availability configured, ghe-config-apply times out while waiting for hookshot-go to start on replica application nodes.

After an administrator enables maintenance mode from the instance's Management Console UI using Firefox, the administrator is redirected to the Settings page, but maintenance mode is not enabled. To work around this issue, use a different browser.

On an instance hosted in AWS, system time may lose synchronization with Amazon's servers after an administrator reboots the instance.

On an instance with the HTTP X-Forwarded-For header configured for use behind a load balancer, all client IP addresses in the instance's audit log erroneously appear as 127.0.0.1.

In GitHub Enterprise Server 3.10 and later, the requirements for TLS security levels have changed due to an upgrade to containers in the underlying OS. On an instance with GitHub Actions enabled and a custom TLS certificate, users may experience disruptions with workflow runs if the TLS certificate uses weak encryption. Workflow runs will not trigger, and the following error message will appear in system logs for babeld.

CA certificate key too weak

To resolve this issue, confirm that your certificate complies with level 2 of the OpenSSL security specification. For more information, see SSL_CTX_set_security_level in the OpenSSL docs. For more information about reviewing your instance's logs, see "About system logs".

If the error appears in babeld logs because your TLS certificate does not comply with level 2 of the specification, you must create and upload a new certificate with stronger security before you upgrade to GitHub Enterprise Server 3.10 or later. For more information, see "Configuring TLS."

On an instance with GitHub Actions enabled, after an upgrade from GitHub Enterprise Server 3.8 or earlier, an internal exception could prevent successful completion of some operations, like upgrades or the configuration of new replica nodes for high availability. If this issue occurs, administrators may see the following error in /data/user/common/ghe-config.log.

Error occurred while executing servicing step 'Clone datatier login to secondary replica' for component CopyAvailabilityGroupSqlLogins during CopyAvailabilityGroupSqlLogins: Object reference not set to an instance of an object.

To resolve this issue, upgrade to the latest patch release of GitHub Enterprise Server.

In some situations, large .adoc files stored in a repository do not render properly in the web UI. The raw contents are still available to view as plaintext.

On an instance in a cluster configuration, restoration of a backup using ghe-restore will exit prematurely if Redis has not restarted properly.

The HAProxy version has been updated in this release. You may encounter elevated error rates when HAProxy is upgraded as part of a hotpatch upgrade to a GitHub Enterprise Server instance. These elevated error rates should resolve within 5 minutes of the hotpatch being applied.

Please note, when performing a hotpatch upgrade to GitHub Enterprise Server version 3.10.4 or higher you will encounter this known issue only if you are hotpatching from GitHub Enterprise Server version 3.10.3 or lower .

If a hotpatch upgrade requires the haproxy-frontend service to be restarted, the restart will hang if there are existing long-lived connections, such as browser web sockets or Git operations. No new connections will be accepted for up to 5 minutes. Any existing unfinished connections at this time will be disconnected. [Updated: 2024-06-17]

HIGH: On an instance with GitHub Connect enabled and non-default settings for GitHub Connect configured, an attacker could use an enterprise GitHub Actions download token to fetch private repository data. This token is only accessible to users on the GitHub Enterprise Server instance. To fix this vulnerability, the Actions download token will now be a permissionless token. GitHub has requested CVE ID CVE-2024-1908 for this vulnerability, which was reported via the GitHub Bug Bounty program.

Packages have been updated to the latest security versions.

Redundant messages caused increased log volumes in /var/log/syslog.

Custom firewall rules are removed during the upgrade process.

During the validation phase of a configuration run, a No such object error may occur for the Notebook and Viewscreen services. This error can be ignored as the services should still correctly start.

If the root site administrator is locked out of the Management Console after failed login attempts, the account does not unlock automatically after the defined lockout time. Someone with administrative SSH access to the instance must unlock the account using the administrative shell. For more information, see "Troubleshooting access to the Management Console."

If an instance is configured to forward logs to a target server with TLS enabled, certificate authority (CA) bundles that a site administrator uploads using ghe-ssl-ca-certificate-install are not respected, and connections to the server fail.

The mbind: Operation not permitted error in the /var/log/mysql/mysql.err file can be ignored. MySQL 8 does not gracefully handle when the CAP_SYS_NICE capability isn't required, and outputs an error instead of a warning.

After an administrator upgrades from GitHub Enterprise Server 3.7 or 3.8, to 3.9 or 3.10, MySQL may not start back up. For more information, see "Known issues with upgrades to your instance."

After an administrator upgrades from GitHub Enterprise Server 3.7 or 3.8 to 3.9 or 3.10, I/O utilization will increase, and in some cases the instance's performance will be impacted. Reduced performance is due to the database server being upgraded from MySQL 5.7 to MySQL 8.0. For more information, see "Known issues with upgrades to your instance."

In rare circumstances, a small instance with both high availability and GitHub Actions configured may report that MSSQL replication is unhealthy after many upgrades with full upgrade packages. If you encounter this issue, contact GitHub Support.

On an instance in a cluster configuration with high availability configured, ghe-config-apply times out while waiting for hookshot-go to start on replica application nodes.

After an administrator enables maintenance mode from the instance's Management Console UI using Firefox, the administrator is redirected to the Settings page, but maintenance mode is not enabled. To work around this issue, use a different browser.

On an instance hosted in AWS, system time may lose synchronization with Amazon's servers after an administrator reboots the instance.

On an instance with the HTTP X-Forwarded-For header configured for use behind a load balancer, all client IP addresses in the instance's audit log erroneously appear as 127.0.0.1.

In GitHub Enterprise Server 3.10 and later, the requirements for TLS security levels have changed due to an upgrade to containers in the underlying OS. On an instance with GitHub Actions enabled and a custom TLS certificate, users may experience disruptions with workflow runs if the TLS certificate uses weak encryption. Workflow runs will not trigger, and the following error message will appear in system logs for babeld.

CA certificate key too weak

To resolve this issue, confirm that your certificate complies with level 2 of the OpenSSL security specification. For more information, see SSL_CTX_set_security_level in the OpenSSL docs. For more information about reviewing your instance's logs, see "About system logs".

If the error appears in babeld logs because your TLS certificate does not comply with level 2 of the specification, you must create and upload a new certificate with stronger security before you upgrade to GitHub Enterprise Server 3.10 or later. For more information, see "Configuring TLS."

On an instance with GitHub Actions enabled, after an upgrade from GitHub Enterprise Server 3.8 or earlier, an internal exception could prevent successful completion of some operations, like upgrades or the configuration of new replica nodes for high availability. If this issue occurs, administrators may see the following error in /data/user/common/ghe-config.log.

Error occurred while executing servicing step 'Clone datatier login to secondary replica' for component CopyAvailabilityGroupSqlLogins during CopyAvailabilityGroupSqlLogins: Object reference not set to an instance of an object.

To resolve this issue, upgrade to the latest patch release of GitHub Enterprise Server.

In some situations, large .adoc files stored in a repository do not render properly in the web UI. The raw contents are still available to view as plaintext.

On an instance in a cluster configuration, restoration of a backup using ghe-restore will exit prematurely if Redis has not restarted properly.

The HAProxy version has been updated in this release. You may encounter elevated error rates when HAProxy is upgraded as part of a hotpatch upgrade to a GitHub Enterprise Server instance. These elevated error rates should resolve within 5 minutes of the hotpatch being applied.

Please note, when performing a hotpatch upgrade to GitHub Enterprise Server version 3.10.4 or higher you will encounter this known issue only if you are hotpatching from GitHub Enterprise Server version 3.10.3 or lower .

If a hotpatch upgrade requires the haproxy-frontend service to be restarted, the restart will hang if there are existing long-lived connections, such as browser web sockets or Git operations. No new connections will be accepted for up to 5 minutes. Any existing unfinished connections at this time will be disconnected. [Updated: 2024-06-17]

HIGH: An attacker could gain unauthorized read permission to files by deploying arbitrary symbolic links to a GitHub Pages site with a specially crafted artifact tarball. GitHub has requested CVE ID CVE-2024-1082 for this vulnerability, which was reported via the GitHub Bug Bounty program.

HIGH: An attacker with an editor role in the Management Console could gain admin SSH access to the appliance by command injection when configuring SAML settings. GitHub has requested CVE ID CVE-2024-1372 for this vulnerability, which was reported via the GitHub Bug Bounty program.

HIGH: An attacker with an editor role in the Management Console could gain admin SSH access to the appliance by command injection when setting an HTTP proxy. GitHub has requested CVE ID CVE-2024-1359 for this vulnerability, which was reported via the GitHub Bug Bounty program.

HIGH: An attacker with an editor role in the Management Console could gain admin SSH access to the appliance by command injection into nomad templates when configuring SMTP options. GitHub has requested CVE ID CVE-2024-1378 for this vulnerability, which was reported via the GitHub Bug Bounty program.

HIGH: An attacker with an editor role in the Management Console could gain admin SSH access to the appliance by command injection in the actions-console docker container while setting a service URL. GitHub has requested CVE ID CVE-2024-1355 for this vulnerability, which was reported via the GitHub Bug Bounty program.

HIGH: An attacker with an editor role in the Management Console could gain admin SSH access to the appliance by command injection in the syslog-ng configuration file. GitHub has requested CVE ID CVE-2024-1354 for this vulnerability, which was reported via the GitHub Bug Bounty program.

HIGH: An attacker with an editor role in the Management Console could gain admin SSH access to the appliance by command injection when setting the username and password for collectd configurations. GitHub has requested CVE ID CVE-2024-1369 for this vulnerability, which was reported via the GitHub Bug Bounty program.

HIGH: An attacker with an editor role in the Management Console could gain admin SSH access to the appliance by command injection into nomad templates when configuring audit log forwarding. GitHub has requested CVE ID CVE-2024-1374 for this vulnerability, which was reported via the GitHub Bug Bounty program.

HIGH: An attacker could create new branches in public repositories, and run arbitrary GitHub Actions workflows with permissions from the GITHUB_TOKEN. GitHub has requested CVE ID CVE-2024-1482 for this vulnerability, which was reported via the GitHub Bug Bounty program.

MEDIUM: An attacker could make changes to a user account by taking advantage of a Cross-site Scripting vulnerability in the tag name pattern field in the tag protections UI. Exploitation of this vulnerability required user interaction with malicious javascript on a website along with further social engineering. GitHub has requested CVE ID CVE-2024-1084 for this vulnerability, which was reported via the GitHub Bug Bounty program.

LOW: An attacker could decrypt the user section of the enterprise user license list JSON file by using an exposed private key. This vulnerability was reported via the GitHub Bug Bounty program.

Packages have been updated to the latest security versions.

On startup, Elasticsearch logged an innocuous JMX MBeans registration error.

Hunk headers in C# files did not correctly display changed functions.

Pre-receive hook failures were not visible in the administrator audit log due to an incomplete bug fix.

When restoring a deleted repository, some metadata associated with the repository, such as packages or project items, did not properly restore.

On an instance with subdomain isolation disabled, a notebook could not be loaded due to an extra / character in the URL path.

During Git data server maintenance, a process that was ran on unsupported GitHub Enterprise Server topologies created a significant amount of system logs but did not perform any repair work.

The default 30 second webhook delivery HTTP timeout can be configured.

Custom firewall rules are removed during the upgrade process.

During the validation phase of a configuration run, a No such object error may occur for the Notebook and Viewscreen services. This error can be ignored as the services should still correctly start.

If the root site administrator is locked out of the Management Console after failed login attempts, the account does not unlock automatically after the defined lockout time. Someone with administrative SSH access to the instance must unlock the account using the administrative shell. For more information, see "Troubleshooting access to the Management Console."

If an instance is configured to forward logs to a target server with TLS enabled, certificate authority (CA) bundles that a site administrator uploads using ghe-ssl-ca-certificate-install are not respected, and connections to the server fail.

The mbind: Operation not permitted error in the /var/log/mysql/mysql.err file can be ignored. MySQL 8 does not gracefully handle when the CAP_SYS_NICE capability isn't required, and outputs an error instead of a warning.

After an administrator upgrades from GitHub Enterprise Server 3.7 or 3.8, to 3.9 or 3.10, MySQL may not start back up. For more information, see "Known issues with upgrades to your instance."

After an administrator upgrades from GitHub Enterprise Server 3.7 or 3.8 to 3.9 or 3.10, I/O utilization will increase, and in some cases the instance's performance will be impacted. Reduced performance is due to the database server being upgraded from MySQL 5.7 to MySQL 8.0. For more information, see "Known issues with upgrades to your instance."

In rare circumstances, a small instance with both high availability and GitHub Actions configured may report that MSSQL replication is unhealthy after many upgrades with full upgrade packages. If you encounter this issue, contact GitHub Support.

On an instance in a cluster configuration with high availability configured, ghe-config-apply times out while waiting for hookshot-go to start on replica application nodes.

After an administrator enables maintenance mode from the instance's Management Console UI using Firefox, the administrator is redirected to the Settings page, but maintenance mode is not enabled. To work around this issue, use a different browser.

On an instance hosted in AWS, system time may lose synchronization with Amazon's servers after an administrator reboots the instance.

On an instance with the HTTP X-Forwarded-For header configured for use behind a load balancer, all client IP addresses in the instance's audit log erroneously appear as 127.0.0.1.

In GitHub Enterprise Server 3.10 and later, the requirements for TLS security levels have changed due to an upgrade to containers in the underlying OS. On an instance with GitHub Actions enabled and a custom TLS certificate, users may experience disruptions with workflow runs if the TLS certificate uses weak encryption. Workflow runs will not trigger, and the following error message will appear in system logs for babeld.

CA certificate key too weak

To resolve this issue, confirm that your certificate complies with level 2 of the OpenSSL security specification. For more information, see SSL_CTX_set_security_level in the OpenSSL docs. For more information about reviewing your instance's logs, see "About system logs".

If the error appears in babeld logs because your TLS certificate does not comply with level 2 of the specification, you must create and upload a new certificate with stronger security before you upgrade to GitHub Enterprise Server 3.10 or later. For more information, see "Configuring TLS."

On an instance with GitHub Actions enabled, after an upgrade from GitHub Enterprise Server 3.8 or earlier, an internal exception could prevent successful completion of some operations, like upgrades or the configuration of new replica nodes for high availability. If this issue occurs, administrators may see the following error in /data/user/common/ghe-config.log.

Error occurred while executing servicing step 'Clone datatier login to secondary replica' for component CopyAvailabilityGroupSqlLogins during CopyAvailabilityGroupSqlLogins: Object reference not set to an instance of an object.

To resolve this issue, upgrade to the latest patch release of GitHub Enterprise Server.

In some situations, large .adoc files stored in a repository do not render properly in the web UI. The raw contents are still available to view as plaintext.

On an instance in a cluster configuration, restoration of a backup using ghe-restore will exit prematurely if Redis has not restarted properly.

The HAProxy version has been updated in this release. You may encounter elevated error rates when HAProxy is upgraded as part of a hotpatch upgrade to a GitHub Enterprise Server instance. These elevated error rates should resolve within 5 minutes of the hotpatch being applied.

Please note, when performing a hotpatch upgrade to GitHub Enterprise Server version 3.10.4 or higher you will encounter this known issue only if you are hotpatching from GitHub Enterprise Server version 3.10.3 or lower .

On an instance with GitHub Actions enabled, Actions workflows that deploy GitHub Pages sites may fail with the following error:

Error: Deployment failed, try again later.

To fix this issue, connect to any of the instance's nodes using SSH, then run the following commands.

if [ -d "$CHROOT_PATH/data/pages-untar" ] ; then
  rm -rf "$CHROOT_PATH/data/pages-untar"
fi
pages_untar_image_tag="$(cat "$CHROOT_PATH/data/docker-image-tags/pages_untar_image_tag")"
id="$(docker create "pages-untar:$pages_untar_image_tag")"
sudo docker cp "$id:/data/pages-untar" "$BASE_PATH/$CHROOT_PATH/data/pages-untar/"
docker rm "$id"
``` [Updated: 2024-03-07]

Redundant messages may cause an increase in the volume of events logged in /var/log/syslog. [Updated: 2024-03-08]

If a hotpatch upgrade requires the haproxy-frontend service to be restarted, the restart will hang if there are existing long-lived connections, such as browser web sockets or Git operations. No new connections will be accepted for up to 5 minutes. Any existing unfinished connections at this time will be disconnected. [Updated: 2024-06-17]

The instance incorrectly wrote the output for multiple workloads to /var/log/syslog.log.

During periods of high traffic, interruptions in service occurred due to insufficient resource allocations for internal components.

When starting up an instance using NVME storage in a cloud other than AWS, the attached data disk was not properly detected.

Custom firewall rules are removed during the upgrade process.

During the validation phase of a configuration run, a No such object error may occur for the Notebook and Viewscreen services. This error can be ignored as the services should still correctly start.

If the root site administrator is locked out of the Management Console after failed login attempts, the account does not unlock automatically after the defined lockout time. Someone with administrative SSH access to the instance must unlock the account using the administrative shell. For more information, see "Troubleshooting access to the Management Console."

If an instance is configured to forward logs to a target server with TLS enabled, certificate authority (CA) bundles that a site administrator uploads using ghe-ssl-ca-certificate-install are not respected, and connections to the server fail.

The mbind: Operation not permitted error in the /var/log/mysql/mysql.err file can be ignored. MySQL 8 does not gracefully handle when the CAP_SYS_NICE capability isn't required, and outputs an error instead of a warning.

After an administrator upgrades from GitHub Enterprise Server 3.7 or 3.8, to 3.9 or 3.10, MySQL may not start back up. For more information, see "Known issues with upgrades to your instance."

After an administrator upgrades from GitHub Enterprise Server 3.7 or 3.8 to 3.9 or 3.10, I/O utilization will increase, and in some cases the instance's performance will be impacted. Reduced performance is due to the database server being upgraded from MySQL 5.7 to MySQL 8.0. For more information, see "Known issues with upgrades to your instance."

In rare circumstances, a small instance with both high availability and GitHub Actions configured may report that MSSQL replication is unhealthy after many upgrades with full upgrade packages. If you encounter this issue, contact GitHub Support.

On an instance in a cluster configuration with high availability configured, ghe-config-apply times out while waiting for hookshot-go to start on replica application nodes.

After an administrator enables maintenance mode from the instance's Management Console UI using Firefox, the administrator is redirected to the Settings page, but maintenance mode is not enabled. To work around this issue, use a different browser.

On an instance hosted in AWS, system time may lose synchronization with Amazon's servers after an administrator reboots the instance.

On an instance with the HTTP X-Forwarded-For header configured for use behind a load balancer, all client IP addresses in the instance's audit log erroneously appear as 127.0.0.1.

In GitHub Enterprise Server 3.10 and later, the requirements for TLS security levels have changed due to an upgrade to containers in the underlying OS. On an instance with GitHub Actions enabled and a custom TLS certificate, users may experience disruptions with workflow runs if the TLS certificate uses weak encryption. Workflow runs will not trigger, and the following error message will appear in system logs for babeld.

CA certificate key too weak

To resolve this issue, confirm that your certificate complies with level 2 of the OpenSSL security specification. For more information, see SSL_CTX_set_security_level in the OpenSSL docs. For more information about reviewing your instance's logs, see "About system logs".

If the error appears in babeld logs because your TLS certificate does not comply with level 2 of the specification, you must create and upload a new certificate with stronger security before you upgrade to GitHub Enterprise Server 3.10 or later. For more information, see "Configuring TLS."

On an instance with GitHub Actions enabled, after an upgrade from GitHub Enterprise Server 3.8 or earlier, an internal exception could prevent successful completion of some operations, like upgrades or the configuration of new replica nodes for high availability. If this issue occurs, administrators may see the following error in /data/user/common/ghe-config.log.

Error occurred while executing servicing step 'Clone datatier login to secondary replica' for component CopyAvailabilityGroupSqlLogins during CopyAvailabilityGroupSqlLogins: Object reference not set to an instance of an object.

To resolve this issue, upgrade to the latest patch release of GitHub Enterprise Server.

In some situations, large .adoc files stored in a repository do not render properly in the web UI. The raw contents are still available to view as plaintext.

Restoring backups with ghe-restore on a GHES cluster will exit prematurely if redis has not restarted properly.

The HAProxy version has been updated in this release. You may encounter elevated error rates when HAProxy is upgraded as part of a hotpatch upgrade to a GitHub Enterprise Server instance. These elevated error rates should resolve within 5 minutes of the hotpatch being applied.

Please note, when performing a hotpatch upgrade to GitHub Enterprise Server version 3.10.4 or higher you will encounter this known issue only if you are hotpatching from GitHub Enterprise Server version 3.10.3 or lower .

On an instance with GitHub Actions enabled, Actions workflows that deploy GitHub Pages sites may fail with the following error:

Error: Deployment failed, try again later.

To fix this issue, connect to any of the instance's nodes using SSH, then run the following commands.

if [ -d "$CHROOT_PATH/data/pages-untar" ] ; then
  rm -rf "$CHROOT_PATH/data/pages-untar"
fi
pages_untar_image_tag="$(cat "$CHROOT_PATH/data/docker-image-tags/pages_untar_image_tag")"
id="$(docker create "pages-untar:$pages_untar_image_tag")"
sudo docker cp "$id:/data/pages-untar" "$BASE_PATH/$CHROOT_PATH/data/pages-untar/"
docker rm "$id"
``` [Updated: 2024-03-07]

Redundant messages may cause an increase in the volume of events logged in /var/log/syslog. [Updated: 2024-03-08]

If a hotpatch upgrade requires the haproxy-frontend service to be restarted, the restart will hang if there are existing long-lived connections, such as browser web sockets or Git operations. No new connections will be accepted for up to 5 minutes. Any existing unfinished connections at this time will be disconnected. [Updated: 2024-06-17]

HIGH: An attacker with access to a Management Console user account with the editor role could escalate privileges through a command injection vulnerability in the Management Console. GitHub has requested CVE ID CVE-2024-0507 for this vulnerability, which was reported via the GitHub Bug Bounty program.

HIGH: An attacker could leverage an unsafe reflection vulnerability in GitHub Enterprise Server (GHES) that could lead to reflection injection. This vulnerability could lead to the execution of user-controlled methods and remote code execution. To exploit this bug, an actor would need to be logged into an account on the GHES instance with the organization owner role. GitHub has requested CVE ID CVE-2024-0200 for this vulnerability, which was reported via the GitHub Bug Bounty program.

Packages have been updated to the latest security versions.

Support for authenticating to GitHub Enterprise Server using GitHub CLI OAuth App with a device code was unintentionally disabled.

During periods of high load, users would see intermittent interruptions to services when upstream services failed internal health checks.

On an instance in a high availability configuration, site administrators using the Manage GitHub Enterprise Server API may have seen a status of UNKNOWN for the MSSQL service.

On an instance with GitHub Actions enabled, some maintenance tasks could fail due to incomplete upgrade steps during previous upgrades to new releases of GitHub Enterprise Server.

Deleting a repository would enqueue unnecessary background jobs that would never complete.

When creating a new custom pattern for secret scanning, the "More options" section of the custom pattern form automatically collapsed when a user entered an invalid regex in the post processing expressions (before/after secret match or additional secret requirements).

On an instance with a GitHub Advanced Security license and secret scanning enabled, users could experience a 500 error when viewing a secret scanning alert page in cases where the alerted commits belonged to the user and one or more commits could not be found.

Members of an enterprise were incorrectly allowed access to the REST API endpoints for Enterprise licensing.

On an instance that uses SAML for authentication, an upgrade from GitHub Enterprise Server 3.7 to 3.9 could result in user login failures due to an outdated gem dependency.

To avoid leaking secrets, the logging of all parameters is disabled for Management Console events in enterprise audit logs.

More detailed information is logged when a GitHub Enterprise Server upgrade failed due to missing database encryption keys.

The branch protection setting to require PR approval of the most recent reviewable push is included in exports from ghe-migrator or the Organization Migrations API.

Custom firewall rules are removed during the upgrade process.

During the validation phase of a configuration run, a No such object error may occur for the Notebook and Viewscreen services. This error can be ignored as the services should still correctly start.

If the root site administrator is locked out of the Management Console after failed login attempts, the account does not unlock automatically after the defined lockout time. Someone with administrative SSH access to the instance must unlock the account using the administrative shell. For more information, see "Troubleshooting access to the Management Console."

If an instance is configured to forward logs to a target server with TLS enabled, certificate authority (CA) bundles that a site administrator uploads using ghe-ssl-ca-certificate-install are not respected, and connections to the server fail.

The mbind: Operation not permitted error in the /var/log/mysql/mysql.err file can be ignored. MySQL 8 does not gracefully handle when the CAP_SYS_NICE capability isn't required, and outputs an error instead of a warning.

After an administrator upgrades from GitHub Enterprise Server 3.7 or 3.8, to 3.9 or 3.10, MySQL may not start back up. For more information, see "Known issues with upgrades to your instance."

After an administrator upgrades from GitHub Enterprise Server 3.7 or 3.8 to 3.9 or 3.10, I/O utilization will increase, and in some cases the instance's performance will be impacted. Reduced performance is due to the database server being upgraded from MySQL 5.7 to MySQL 8.0. For more information, see "Known issues with upgrades to your instance."

In rare circumstances, a small instance with both high availability and GitHub Actions configured may report that MSSQL replication is unhealthy after many upgrades with full upgrade packages. If you encounter this issue, contact GitHub Support.

On an instance in a cluster configuration with high availability configured, ghe-config-apply times out while waiting for hookshot-go to start on replica application nodes.

After an administrator enables maintenance mode from the instance's Management Console UI using Firefox, the administrator is redirected to the Settings page, but maintenance mode is not enabled. To work around this issue, use a different browser.

On an instance hosted in AWS, system time may lose synchronization with Amazon's servers after an administrator reboots the instance.

On an instance with the HTTP X-Forwarded-For header configured for use behind a load balancer, all client IP addresses in the instance's audit log erroneously appear as 127.0.0.1.

In GitHub Enterprise Server 3.10 and later, the requirements for TLS security levels have changed due to an upgrade to containers in the underlying OS. On an instance with GitHub Actions enabled and a custom TLS certificate, users may experience disruptions with workflow runs if the TLS certificate uses weak encryption. Workflow runs will not trigger, and the following error message will appear in system logs for babeld.

CA certificate key too weak

To resolve this issue, confirm that your certificate complies with level 2 of the OpenSSL security specification. For more information, see SSL_CTX_set_security_level in the OpenSSL docs. For more information about reviewing your instance's logs, see "About system logs".

If the error appears in babeld logs because your TLS certificate does not comply with level 2 of the specification, you must create and upload a new certificate with stronger security before you upgrade to GitHub Enterprise Server 3.10 or later. For more information, see "Configuring TLS."

On an instance with GitHub Actions enabled, after an upgrade from GitHub Enterprise Server 3.8 or earlier, an internal exception could prevent successful completion of some operations, like upgrades or the configuration of new replica nodes for high availability. If this issue occurs, administrators may see the following error in /data/user/common/ghe-config.log.

Error occurred while executing servicing step 'Clone datatier login to secondary replica' for component CopyAvailabilityGroupSqlLogins during CopyAvailabilityGroupSqlLogins: Object reference not set to an instance of an object.

To resolve this issue, upgrade to the latest patch release of GitHub Enterprise Server.

In some situations, large .adoc files stored in a repository do not render properly in the web UI. The raw contents are still available to view as plaintext.

Restoring backups with ghe-restore on a GHES cluster will exit prematurely if redis has not restarted properly.

The HAProxy version has been updated in this release. You may encounter elevated error rates when HAProxy is upgraded as part of a hotpatch upgrade to a GitHub Enterprise Server instance. These elevated error rates should resolve within 5 minutes of the hotpatch being applied.

Please note, when performing a hotpatch upgrade to GitHub Enterprise Server version 3.10.4 or higher you will encounter this known issue only if you are hotpatching from GitHub Enterprise Server version 3.10.3 or lower .

During periods of high traffic, interruptions in service may occur due to insufficient resource allocations for internal components. [Updated 2024-01-23]

The settings for enabling scheduled reminders were added unintentionally to this release. Scheduled reminders are not officially supported. [Updated: 2024-02-22]

Redundant messages may cause an increase in the volume of events logged in /var/log/syslog. [Updated: 2024-03-08]

If a hotpatch upgrade requires the haproxy-frontend service to be restarted, the restart will hang if there are existing long-lived connections, such as browser web sockets or Git operations. No new connections will be accepted for up to 5 minutes. Any existing unfinished connections at this time will be disconnected. [Updated: 2024-06-17]

These release notes previously indicated that GitHub Enterprise Server 3.10.5 contained fixes for the following issues:

• An improper authentication vulnerability that affected private mode, CVE-2023-6847
• An incorrect authorization vulnerability that affected issue comments, CVE-2023-51380

These fixes were included in GitHub Enterprise Server 3.10.4.

HIGH: An improper authentication vulnerability was identified in GitHub Enterprise Server that allowed a bypass of private mode by using a specially crafted API request. Private mode is the mechanism that enforces authentication for publicly-scoped resources. For more information, see "Enabling private mode."

This vulnerability would allow unauthenticated attackers to gain access to various types of resources set as public on the instance. To exploit this vulnerability, an attacker would need network access to the GitHub Enterprise Server instance configured in private mode. This vulnerability was reported via the GitHub Bug Bounty program and assigned CVE-2023-6847.

HIGH: An attacker with access to a Management Console user account with the editor role could escalate privileges by making requests to the endpoint used for bootstrapping the instance, and then reset the root site administrator password. GitHub has requested CVE ID CVE-2023-46647 for this vulnerability, which was reported via the GitHub Bug Bounty program.

HIGH: A path traversal vulnerability was identified in GitHub Enterprise Server that allowed arbitrary file reading when building a GitHub Pages site. To exploit this vulnerability, an attacker would need permission to create and build a GitHub Pages site on the GitHub Enterprise Server instance. This vulnerability was reported via the GitHub Bug Bounty program and assigned CVE-2023-46645.

MEDIUM: An insertion of sensitive information into log file vulnerability was identified in the log files for a GitHub Enterprise Server backend service that could permit an adversary in the middle attack when combined with other phishing techniques. To exploit this, an attacker would need access to the log files for the GitHub Enterprise Server instance, a backup archive created with GitHub Enterprise Server Backup Utilities, or a service which received streamed logs. GitHub has requested CVE ID CVE-2023-6746 for this vulnerability.

MEDIUM: Due to an insufficient entropy vulnerability, an attacker could brute force a user invitation to the Management Console. To exploit this vulnerability, an attacker would have needed knowledge that a user invitation was pending. This vulnerability was reported via the GitHub Bug Bounty program and assigned CVE-2023-46648.

MEDIUM: An attacker could maintain admin access via a race condition when an organization was converted from a user. GitHub has requested CVE ID CVE-2023-46649 for this vulnerability, which was reported via the GitHub Bug Bounty program.

MEDIUM: Due to an improper access control, an attacker could view private repository names by enumerating check run IDs with the "Get a check run" API endpoint. This vulnerability did not allow unauthorized access to any repository content other than the name. GitHub has requested CVE ID CVE-2023-46646 for this vulnerability, which was reported via the GitHub Bug Bounty program.

MEDIUM: An attacker could maintain admin access to a transferred repository in a race condition by making a GraphQL mutation to alter repository permissions during the transfer. GitHub has requested CVE ID CVE-2023-6690 for this vulnerability, which reported via the GitHub Bug Bounty program.

MEDIUM: An insertion of sensitive information into log file in the audit log in GitHub Enterprise Server was identified that that could allow an attacker to gain access to the Management Console. To exploit this, an attacker would need access to the log files for the GitHub Enterprise Server instance, a backup archive created with GitHub Enterprise Server Backup Utilities, or a service which received streamed logs. GitHub has requested CVE ID CVE-2023-6802 for this vulnerability.

MEDIUM: A race condition in GitHub Enterprise Server allowed an outside collaborator to be added while a repository is being transferred. GitHub has requested CVE ID CVE-2023-6803 for this vulnerability, which was reported via the GitHub Bug Bounty program.

MEDIUM: Improper privilege management allowed arbitrary workflows to be committed and run using an improperly scoped personal access token. To exploit this, a workflow must have already existed in the target repository. GitHub has requested CVE ID CVE-2023-6804 for this vulnerability, which was reported via the GitHub Bug Bounty program.

MEDIUM: An incorrect authorization vulnerability was identified that allowed issue comments to be updated with an improperly scoped token. This vulnerability did not allow unauthorized access to any repository content as it also required contents.write and issues.read permissions. This vulnerability was reported via the GitHub Bug Bounty Program and has been assigned CVE-2023-51379.

MEDIUM: An incorrect authorization vulnerability was identified that allowed issue comments to be read with an improperly scoped token. This vulnerability was reported via the GitHub Bug Bounty Program and has been assigned CVE-2023-51380.

LOW: Pre-receive hooks have been further hardened against shell command injections.

LOW: To render interactive maps in an instance's web UI using Azure Maps, GitHub Enterprise Server has migrated from use of an unsecure Azure Maps API token to a more secure access token provided by role-based access control (RBAC) in Entra ID. After upgrading to this release, to re-enable interactive maps, an administrator must reconfigure authentication to Azure Maps in the Management Console. For more information, see "Configuring interactive maps."

To address scenarios that could lead to denial of service, HAProxy has been upgraded to version 2.8.4.

Packages have been updated to the latest security versions.

In rare cases, on an instance with GitHub Actions enabled, a failed check on a deleted repository could cause upgrades to a new version of GitHub Enterprise Server to fail.

On an instance in a high availability configuration, site administrators using the ghe-repl-status command-line utility may have seen a status of UNKNOWN for the MSSQL service.

When an administrator ran the ghe-support-bundle or ghe-cluster-support-bundle command, the -p flag did not produce bundles with log durations as specified. The duration period can now only be specified in days. Additionally, unnecessary files were sanitized by the commands.

On an instance with GitHub Actions enabled, some maintenance tasks could fail due to incomplete upgrade steps during previous upgrades to new releases of GitHub Enterprise Server.

On an instance in a cluster configuration, site administrators using the ghe-config-apply utility may have seen the extraneous message "Error: Server closed the connection" in the logs for the utility.

Some OAuth applications did not have device code flow (DCF) explicitly enabled, which prevented DCF from running correctly.

On an instance in a cluster configuration, upgrades could fail due to a background job running during database migration.

On an instance with a GitHub Advanced Security license and secret scanning enabled, site administrators using the ghe-secret-scanning command would not see a relevant error message if their input was invalid.

On an instance in a high availability configuration, the ghe-repl-teardown command failed when provided with a UUID.

Support for authenticating to GitHub Enterprise Server from Visual Studio Code with a device code was unintentionally disabled.

In some environments, stale .backup log files could accumulate in the system.

On an instance hosted on AWS, when configuring GitHub Packages, virtual-hosted-style AWS S3 URLs would default to path-style URLs if a region-code was included. For more information, see Virtual hosting of buckets in the AWS documentation.

In some cases, when an administrator uploaded a custom TLS certificate, the certificate was not correctly installed on the instance.

Because the | character was not permitted, administrators could not add an SMTP username to authenticate with the Azure Communication Service.

On an instance with a GitHub Advanced Security license, users with the security manager role could not update custom links for push protection using the REST API.

On an instance with the dependency graph enabled, some security products were not automatically enabled for new public repositories.

Pull request review threads at the file level, rather than the individual line level, were not included in exports from ghe-migrator or the Organization Migrations API.

Some responses from the REST API included an incorrect URL in the link header.

After importing a migration archive using ghe-migrator or REST API endpoints for organization migrations, in some cases, some review comments within pull requests were not associated with lines of code.

In some cases, /stafftools/users/:login incorrectly displayed trade screening information.

Deprecated resource_activity jobs were not processed and accumulated over time in the queue, causing possible memory issues.

On an instance with a GitHub Advanced Security license and secret scanning enabled, secret scanning alert emails were sent to organization owners even if their email address did not comply with domain restrictions.

After a user started a repository transfer, if another user viewed the repository before the transfer finished, the repository overview rendered incorrectly.

On an instance with GitHub Connect and unified search enabled, users trying to view the unified search code results would get a 500 error.

On some instances, the user interface displayed a codespace button, even though GitHub Codespaces is not supported on GitHub Enterprise Server.

On an instance with GitHub Actions enabled, users occasionally got a 500 error when viewing a job with a pending deployment.

On an instance with GitHub Actions enabled, an issue with GH_TOKEN sometimes prevented GitHub Pages sites from building successfully in workflows.

An administrator could enable GitHub Connect on an instance with a license that does not support GitHub Connect.

On an instance with GitHub Connect enabled, some system users were incorrectly counted as consuming a license following license sync.

The enterprise account pages on some installations rendered very slowly.

A user in the process of being converted into an organization could be added as a collaborator on a repository. This resulted in the new organizations owners unexpectedly receiving access to the repository.

When using ghe-migrator to import repositories into GitHub Enterprise Server, the conflicts and audit subcommands produced an invalid CSV file due to an extra log line appended to the file.

Pre-receive hook failures were not visible in the administrator audit log.

Running ghe-spokesctl gov info without any arguments caused a panic response.

On an instance with a GitHub Advanced Security license and secret scanning enabled, dry runs sometimes incorrectly reported no results for custom patterns.

On an instance with a GitHub Advanced Security license and secret scanning enabled, webhooks for alert locations did not contain information about push protection bypasses.

On an instance with a GitHub Advanced Security license, code scanning would report an incorrect number of files scanned on the "Tools" status page.

On an instance with Dependabot updates enabled, Dependabot relies on the node installation provided by the actions runner instead of dynamically downloading.

When adding a node to an instance, performance is improved during initial database replication.

An administrator can run the new ghe-check-background-upgrade-jobs command to ensure all upgrade jobs that run in the background have finished. This allows the administrator to know when they can start the next upgrade to their GitHub Enterprise Server instance.

To avoid negative effects on disk utilization, babeld log files have a maximum size of 15 GB.

To improve reliability of release uploads in low-bandwidth environments, the time-to-live (TTL) value of the token for uploading release assets has increased from 1 hour to 3 hours.

When using ghe-migrator prepare to import an archive, a missing schema.json file results in an UnsupportedArchive error rather than an UnsupportedSchemaVersion error.

The audit log now tracks all failed password attempts individually. Previously, duplicate failed password attempts in sequence within the same day would be grouped into one failed password attempt, with a count field.

Custom firewall rules are removed during the upgrade process.

The GitHub Packages npm registry no longer returns a time value in metadata responses. This was done to allow for substantial performance improvements. We continue to have all the data necessary to return a time value as part of the metadata response and will resume returning this value in the future once we have solved the existing performance issues.

During the validation phase of a configuration run, a No such object error may occur for the Notebook and Viewscreen services. This error can be ignored as the services should still correctly start.

If the root site administrator is locked out of the Management Console after failed login attempts, the account does not unlock automatically after the defined lockout time. Someone with administrative SSH access to the instance must unlock the account using the administrative shell. For more information, see "Troubleshooting access to the Management Console."

If an instance is configured to forward logs to a target server with TLS enabled, certificate authority (CA) bundles that a site administrator uploads using ghe-ssl-ca-certificate-install are not respected, and connections to the server fail.

The mbind: Operation not permitted error in the /var/log/mysql/mysql.err file can be ignored. MySQL 8 does not gracefully handle when the CAP_SYS_NICE capability isn't required, and outputs an error instead of a warning.

After an administrator upgrades from GitHub Enterprise Server 3.7 or 3.8, to 3.9 or 3.10, MySQL may not start back up. For more information, see "Known issues with upgrades to your instance."

After an administrator upgrades from GitHub Enterprise Server 3.7 or 3.8 to 3.9 or 3.10, I/O utilization will increase, and in some cases the instance's performance will be impacted. Reduced performance is due to the database server being upgraded from MySQL 5.7 to MySQL 8.0. For more information, see "Known issues with upgrades to your instance."

In rare circumstances, a small instance with both high availability and GitHub Actions configured may report that MSSQL replication is unhealthy after many upgrades with full upgrade packages. If you encounter this issue, contact GitHub Support.

On an instance in a cluster configuration with high availability configured, ghe-config-apply times out while waiting for hookshot-go to start on replica application nodes.

After an administrator enables maintenance mode from the instance's Management Console UI using Firefox, the administrator is redirected to the Settings page, but maintenance mode is not enabled. To work around this issue, use a different browser.

On an instance hosted in AWS, system time may lose synchronization with Amazon's servers after an administrator reboots the instance.

On an instance with the HTTP X-Forwarded-For header configured for use behind a load balancer, all client IP addresses in the instance's audit log erroneously appear as 127.0.0.1.

In GitHub Enterprise Server 3.10 and later, the requirements for TLS security levels have changed due to an upgrade to containers in the underlying OS. On an instance with GitHub Actions enabled and a custom TLS certificate, users may experience disruptions with workflow runs if the TLS certificate uses weak encryption. Workflow runs will not trigger, and the following error message will appear in system logs for babeld.

CA certificate key too weak

To resolve this issue, confirm that your certificate complies with level 2 of the OpenSSL security specification. For more information, see SSL_CTX_set_security_level in the OpenSSL docs. For more information about reviewing your instance's logs, see "About system logs".

If the error appears in babeld logs because your TLS certificate does not comply with level 2 of the specification, you must create and upload a new certificate with stronger security before you upgrade to GitHub Enterprise Server 3.10 or later. For more information, see "Configuring TLS."

On an instance with GitHub Actions enabled, after an upgrade from GitHub Enterprise Server 3.8 or earlier, an internal exception could prevent successful completion of some operations, like upgrades or the configuration of new replica nodes for high availability. If this issue occurs, administrators may see the following error in /data/user/common/ghe-config.log.

Error occurred while executing servicing step 'Clone datatier login to secondary replica' for component CopyAvailabilityGroupSqlLogins during CopyAvailabilityGroupSqlLogins: Object reference not set to an instance of an object.

To resolve this issue, upgrade to the latest patch release of GitHub Enterprise Server.

In some situations, large .adoc files stored in a repository do not render properly in the web UI. The raw contents are still available to view as plaintext.

Restoring backups with ghe-restore on a GHES cluster will exit prematurely if redis has not restarted properly.

The HAProxy version has been updated in this release. You may encounter elevated error rates when HAProxy is upgraded as part of a hotpatch upgrade to a GitHub Enterprise Server instance. These elevated error rates should resolve within 5 minutes of the hotpatch being applied.

Please note, when performing a hotpatch upgrade to GitHub Enterprise Server version 3.10.4 or higher you will encounter this known issue only if you are hotpatching from GitHub Enterprise Server version 3.10.3 or lower . [Updated 2024-01-03]

During periods of high traffic, interruptions in service may occur due to insufficient resource allocations for internal components. [Updated 2024-01-23]

The settings for enabling scheduled reminders were added unintentionally to this release. Scheduled reminders are not officially supported. [Updated: 2024-02-22]

Redundant messages may cause an increase in the volume of events logged in /var/log/syslog. [Updated: 2024-03-08]

If a hotpatch upgrade requires the haproxy-frontend service to be restarted, the restart will hang if there are existing long-lived connections, such as browser web sockets or Git operations. No new connections will be accepted for up to 5 minutes. Any existing unfinished connections at this time will be disconnected. [Updated: 2024-06-17]

Interactive maps in the web UI no longer allow authentication using an Azure Maps API key

• To allow users to render interactive maps in an instance's web UI by writing GeoJSON or TopoJSON syntax, GitHub Enterprise Server previously required a potentially unsecure API key for authentication with Azure Maps. If an administrator previously enabled interactive maps on an instance, the feature is disabled upon upgrade to this release.

  To re-enable interactive maps for your instance, you must configure an application on an Entra ID tenant that has access to Azure Maps using role-based access control (RBAC). For more information, see "Configuring interactive maps" and the security fixes for this release.

LOW: Due to an incorrect permission assignment for some configuration files, an attacker with access to a local operating system user account could read MySQL connection details including the MySQL password. [Updated: 2023-11-13]

Packages have been updated to the latest security versions.

Authentication of programmatic access tokens did not fully validate the status of token's users, which allowed token authentication requests to succeed even if the associated user was not allowed to make such requests. This issue is unrelated to validation of token scope.

The dependency-graph-api service sometimes rapidly filled logs with a large amount of Base64-encoded response data, particularly during upgrades.

After an administrator made changes to maintenance mode from the instance's Management Console UI using Firefox, the administrator was redirected to the Settings page, but their changes were not enabled.

The ghe-cluster-repl-status command did not display all replication statuses.

On an instance in a cluster configuration with high availability enabled, ghe-config-apply timed out while waiting for hookshot-go to start on replica application nodes.

/var/log/lastlog was not copied over as a sparse file during ghe-upgrade, which could cause issues by using additional disk space.

On an instance in a cluster configuration, when managing maintenance mode using ghe-cluster-maintenance, an erroneous warning appeared that read "Warning: Maintenance mode set on primary, please make sure to set it on any active replica if needed".

ghe-repl-status did not identify Git replicas in certain incomplete states and incorrectly suggested that a failover could be performed safely. In some cases, this led to data loss during failover.

Repository exports using ghe-migrator or the REST API's operation for organization migrations could fail when a large number of commit comments or long commit comments were present.

On an instance with a GitHub Advanced Security license and secret scanning enabled, secret scanning suggested incorrect filters when viewing both open and closed alerts.

On instances using the private beta of SCIM provisioning, some users were presented with a "single sign-in" hover card.

On an instance with multiple nodes, ghe-spokes status did not identify Git replicas in certain incomplete states, causing a false report that replication was in sync and leading to data loss or replication issues during failover.

On an instance with GitHub Actions enabled, administrators received a 500 error after attempting to force cancel a workflow run via Staff Tools.

On an instance with a GitHub Advanced Security license, repositories within organizations created using the + dropdown menu did not have GitHub Advanced Security features enabled automatically, even if the features should have been enabled.

On an instance with a GitHub Advanced Security license and secret scanning enabled, dry runs sometimes incorrectly reported no results for custom patterns.

Instructions in the "Migrations" section of the Management Console clarify that only standard AWS S3 endpoints are supported when configuring AWS S3 as a blob storage provider for migrations.

On an instance in a cluster configuration, administrators can identify the repository networks or gists that are common across a specified set of storage nodes using the spokesctl find-on-replicas command.

As a security measure, GitHub Pages does not build sites that contain symbolic links except when using custom GitHub Actions workflows. This change strengthens GitHub Pages's symbolic link detection.

In GitHub Enterprise Server 3.10 and later, the requirements for TLS security levels have changed due to an upgrade to containers in the underlying OS. On an instance with GitHub Actions enabled and a custom TLS certificate, users may experience disruptions with workflow runs if the TLS certificate uses weak encryption. Workflow runs will not trigger, and the following error message will appear in system logs for babeld.

CA certificate key too weak

To resolve this issue, confirm that your certificate complies with level 2 of the OpenSSL security specification. For more information, see SSL_CTX_set_security_level in the OpenSSL docs. For more information about reviewing your instance's logs, see "About system logs".

If the error appears in babeld logs because your TLS certificate does not comply with level 2 of the specification, you must create and upload a new certificate with stronger security before you upgrade to GitHub Enterprise Server 3.10 or later. For more information, see "Configuring TLS." [Updated: 2023-10-26]

Custom firewall rules are removed during the upgrade process.

During the validation phase of a configuration run, a No such object error may occur for the Notebook and Viewscreen services. This error can be ignored as the services should still correctly start.

If the root site administrator is locked out of the Management Console after failed login attempts, the account does not unlock automatically after the defined lockout time. Someone with administrative SSH access to the instance must unlock the account using the administrative shell. For more information, see "Troubleshooting access to the Management Console."

On an instance in a high-availability configuration, passive replica nodes accept Git client requests and forward the requests to the primary node.

If an instance is configured to forward logs to a target server with TLS enabled, certificate authority (CA) bundles that a site administrator uploads using ghe-ssl-ca-certificate-install are not respected, and connections to the server fail.

When running ghe-config-apply, the process may stall with the message Deployment is running pending automatic promotion.

The mbind: Operation not permitted error in the /var/log/mysql/mysql.err file can be ignored. MySQL 8 does not gracefully handle when the CAP_SYS_NICE capability isn't required, and outputs an error instead of a warning.

After an administrator upgrades from GitHub Enterprise Server 3.7 or 3.8, to 3.9 or 3.10, MySQL may not start back up. For more information, see "Known issues with upgrades to your instance." [Updated: 2023-08-11]

After an administrator upgrades from GitHub Enterprise Server 3.7 or 3.8 to 3.9 or 3.10, I/O utilization will increase, and in some cases the instance's performance will be impacted. Reduced performance is due to the database server being upgraded from MySQL 5.7 to MySQL 8.0. For more information, see "Known issues with upgrades to your instance."

In rare circumstances, a small instance with both high availability and GitHub Actions configured may report that MSSQL replication is unhealthy after many upgrades with full upgrade packages. If you encounter this issue, contact GitHub Support.

On an instance in a cluster configuration with high availability configured, ghe-config-apply times out while waiting for hookshot-go to start on replica application nodes.

After an administrator enables maintenance mode from the instance's Management Console UI using Firefox, the administrator is redirected to the Settings page, but maintenance mode is not enabled. To work around this issue, use a different browser.

When an administrator uses the -p flag with the ghe-support-bundle utility to collect data for a specific number of hours, the utility erroneously collects more logs than necessary.

Jobs in a deprecated queue are not processed and may accumulate over time. These jobs are reflected in the monitor dashboard's "Aqueduct queue depth" graph as an increase in resource_activity. In some cases, a buildup of unprocessed jobs can result in memory exhaustion. If you observe memory exhaustion on your instance and see a high metric for resource_activity, contact GitHub Support. [Updated: 2023-10-26]

On an instance with GitHub Actions enabled, after an upgrade from GitHub Enterprise Server 3.8 or earlier, an internal exception could prevent successful completion of some operations, like upgrades or the configuration of new replica nodes for high availability. If this issue occurs, administrators may see the following error in /data/user/common/ghe-config.log.

Error occurred while executing servicing step 'Clone datatier login to secondary replica' for component CopyAvailabilityGroupSqlLogins during CopyAvailabilityGroupSqlLogins: Object reference not set to an instance of an object.

To resolve this issue, upgrade to the latest patch release of GitHub Enterprise Server. [Updated: 2023-12-04]

In some situations, large .adoc files stored in a repository do not render properly in the web UI. The raw contents are still available to view as plaintext. [Updated: 2023-10-31]

On an instance hosted in AWS, system time may lose synchronization with Amazon's servers after an administrator reboots the instance. [Updated 2023-11-10]

On an instance in a cluster configuration, restoration of a backup using ghe-restore will exit prematurely if Redis has not restarted properly. [Updated 2023-12-05]

On an instance with the HTTP X-Forwarded-For header configured for use behind a load balancer, all client IP addresses in the instance's audit log erroneously appear as 127.0.0.1. [Updated 2023-12-13]

During periods of high traffic, interruptions in service may occur due to insufficient resource allocations for internal components. [Updated 2024-01-23]

The settings for enabling scheduled reminders were added unintentionally to this release. Scheduled reminders are not officially supported. [Updated: 2024-02-22]

Redundant messages may cause an increase in the volume of events logged in /var/log/syslog. [Updated: 2024-03-08]

If a hotpatch upgrade requires the haproxy-frontend service to be restarted, the restart will hang if there are existing long-lived connections, such as browser web sockets or Git operations. No new connections will be accepted for up to 5 minutes. Any existing unfinished connections at this time will be disconnected. [Updated: 2024-06-17]

On an instance in a high-availability, geo-replication, or repository cache configuration, prolonged replication issues could occur on replica nodes due to failure of SpokesRepairRepoReplicaJob and SpokesSyncCacheReplicaJob jobs.

In GitHub Enterprise Server 3.10 and later, the requirements for TLS security levels have changed due to an upgrade to containers in the underlying OS. On an instance with GitHub Actions enabled and a custom TLS certificate, users may experience disruptions with workflow runs if the TLS certificate uses weak encryption. Workflow runs will not trigger, and the following error message will appear in system logs for babeld.

CA certificate key too weak

To resolve this issue, confirm that your certificate complies with level 2 of the OpenSSL security specification. For more information, see SSL_CTX_set_security_level in the OpenSSL docs. For more information about reviewing your instance's logs, see "About system logs".

If the error appears in babeld logs because your TLS certificate does not comply with level 2 of the specification, you must create and upload a new certificate with stronger security before you upgrade to GitHub Enterprise Server 3.10 or later. For more information, see "Configuring TLS." [Updated: 2023-10-26]

Custom firewall rules are removed during the upgrade process.

During the validation phase of a configuration run, a No such object error may occur for the Notebook and Viewscreen services. This error can be ignored as the services should still correctly start.

If the root site administrator is locked out of the Management Console after failed login attempts, the account does not unlock automatically after the defined lockout time. Someone with administrative SSH access to the instance must unlock the account using the administrative shell. For more information, see "Troubleshooting access to the Management Console." [Updated: 2023-02-23]

On an instance in a high-availability configuration, passive replica nodes accept Git client requests and forward the requests to the primary node.

If an instance is configured to forward logs to a target server with TLS enabled, certificate authority (CA) bundles that a site administrator uploads using ghe-ssl-ca-certificate-install are not respected, and connections to the server fail.

When running ghe-config-apply, the process may stall with the message Deployment is running pending automatic promotion.

The mbind: Operation not permitted error in the /var/log/mysql/mysql.err file can be ignored. MySQL 8 does not gracefully handle when the CAP_SYS_NICE capability isn't required, and outputs an error instead of a warning.

After an administrator upgrades from GitHub Enterprise Server 3.7 or 3.8, to 3.9 or 3.10, MySQL may not start back up. For more information, see "Known issues with upgrades to your instance." [Updated: 2023-08-11]

After an administrator upgrades from GitHub Enterprise Server 3.7 or 3.8 to 3.9 or 3.10, I/O utilization will increase, and in some cases the instance's performance will be impacted. Reduced performance is due to the database server being upgraded from MySQL 5.7 to MySQL 8.0. For more information, see "Known issues with upgrades to your instance."

In rare circumstances, a small instance with both high availability and GitHub Actions configured may report that MSSQL replication is unhealthy after many upgrades with full upgrade packages. If you encounter this issue, contact GitHub Support.

On an instance in a cluster configuration with high availability configured, ghe-config-apply times out while waiting for hookshot-go to start on replica application nodes.

After an administrator enables maintenance mode from the instance's Management Console UI using Firefox, the administrator is redirected to the Settings page, but maintenance mode is not enabled. To work around this issue, use a different browser.

When an administrator uses the -p flag with the ghe-support-bundle utility to collect data for a specific number of hours, the utility erroneously collects more logs than necessary. [Updated: 2023-10-13]

The settings for enabling scheduled reminders were added unintentionally to this release. Scheduled reminders are not officially supported. [Updated: 2023-10-17]

Jobs in a deprecated queue are not processed and may accumulate over time. These jobs are reflected in the monitor dashboard's "Aqueduct queue depth" graph as an increase in resource_activity. In some cases, a buildup of unprocessed jobs can result in memory exhaustion. If you observe memory exhaustion on your instance and see a high metric for resource_activity, contact GitHub Support. [Updated: 2023-10-26]

On an instance with GitHub Actions enabled, after an upgrade from GitHub Enterprise Server 3.8 or earlier, an internal exception could prevent successful completion of some operations, like upgrades or the configuration of new replica nodes for high availability. If this issue occurs, administrators may see the following error in /data/user/common/ghe-config.log.

Error occurred while executing servicing step 'Clone datatier login to secondary replica' for component CopyAvailabilityGroupSqlLogins during CopyAvailabilityGroupSqlLogins: Object reference not set to an instance of an object.

To resolve this issue, upgrade to the latest patch release of GitHub Enterprise Server. [Updated: 2023-12-04]

In some situations, large .adoc files stored in a repository do not render properly in the web UI. The raw contents are still available to view as plaintext. [Updated: 2023-10-31]

On an instance hosted in AWS, system time may lose synchronization with Amazon's servers after an administrator reboots the instance. [Updated 2023-11-10]

On an instance in a cluster configuration, restoration of a backup using ghe-restore will exit prematurely if Redis has not restarted properly. [Updated 2023-12-05]

On an instance with the HTTP X-Forwarded-For header configured for use behind a load balancer, all client IP addresses in the instance's audit log erroneously appear as 127.0.0.1. [Updated 2023-12-13]

During periods of high traffic, interruptions in service may occur due to insufficient resource allocations for internal components. [Updated 2024-01-23]

Redundant messages may cause an increase in the volume of events logged in /var/log/syslog. [Updated: 2024-03-08]

If a hotpatch upgrade requires the haproxy-frontend service to be restarted, the restart will hang if there are existing long-lived connections, such as browser web sockets or Git operations. No new connections will be accepted for up to 5 minutes. Any existing unfinished connections at this time will be disconnected. [Updated: 2024-06-17]

HTTP Strict Transport Security (HSTS) is enabled within the Management Console.

Packages have been updated to the latest security versions.

LOW: An incorrect comparison vulnerability was identified in GitHub Enterprise Server that allowed commit smuggling by displaying an incorrect diff in a reopened pull request. To exploit this vulnerability, an attacker would need write access to the repository. This vulnerability was reported via the GitHub Bug Bounty program and was assigned CVE-2023-23766. [Updated: 2023-09-22]

On an instance with GitHub Actions enabled, scale sets configured at the enterprise level did not appear for use within the instance's organizations or repositories.

On an instance with a GitHub Advanced Security license and secret scanning enabled, secret scanning alerts could fail to show an error message in the UI when a failure occurred closing or reopening the alert.

On an instance with a GitHub Advanced Security license and secret scanning enabled, and when using Safari, changing additional match requirements for a custom pattern did not retrigger custom pattern evaluation against a user submitted test string.

On an instance with a GitHub Advanced Security license and secret scanning enabled, organization access for a leaked GitHub tokens was not shown to commit authors when viewing the alert.

On an instance with a GitHub Advanced Security license and secret scanning enabled, when token location(s) included a commit that introduced a large change, the page for viewing the alert would load slowly.

When uploading migration archives to blob storage, the GitHub Enterprise Server instance's outbound web proxy server was not used.

On an enterprise with the policy setting that disallows repository admins from enabling/disabling secret scanning, transferring a repository to a new organization that automatically enabled secret scanning wouldn't result in the transferred repository being automatically enabled for secret scanning.

When migrating a repository from a GitHub Enterprise Server instance to another location, the ghe-migrator target_url command allows you to record the repository's new location. The new URL is displayed when you visit the main page of the repository in the web interface.

On an instance with subdomain isolation disabled, a notebook could not be loaded due to incorrect asset paths.

On an instance with a GitHub Advanced Security license and secret scanning enabled, in some cases, custom patterns would erroneously show no results for a dry run.

When GitHub Enterprise checks for a new upgrade or hotpatch package, if the check fails the failure details are output to the ghe-update-check log, and the Management Console UI provides a "Check Again" button to rerun the check.

When providing data to GitHub Support, GitHub Enterprise Server displays a notice describing how support data is used before uploading the support files.

In GitHub Enterprise Server 3.10 and later, the requirements for TLS security levels have changed due to an upgrade to containers in the underlying OS. On an instance with GitHub Actions enabled and a custom TLS certificate, users may experience disruptions with workflow runs if the TLS certificate uses weak encryption. Workflow runs will not trigger, and the following error message will appear in system logs for babeld.

CA certificate key too weak

To resolve this issue, confirm that your certificate complies with level 2 of the OpenSSL security specification. For more information, see SSL_CTX_set_security_level in the OpenSSL docs. For more information about reviewing your instance's logs, see "About system logs".

If the error appears in babeld logs because your TLS certificate does not comply with level 2 of the specification, you must create and upload a new certificate with stronger security before you upgrade to GitHub Enterprise Server 3.10 or later. For more information, see "Configuring TLS." [Updated: 2023-10-26]

Custom firewall rules are removed during the upgrade process.

During the validation phase of a configuration run, a No such object error may occur for the Notebook and Viewscreen services. This error can be ignored as the services should still correctly start.

If the root site administrator is locked out of the Management Console after failed login attempts, the account does not unlock automatically after the defined lockout time. Someone with administrative SSH access to the instance must unlock the account using the administrative shell. For more information, see "Troubleshooting access to the Management Console." [Updated: 2023-02-23]

On an instance in a high-availability configuration, passive replica nodes accept Git client requests and forward the requests to the primary node.

If an instance is configured to forward logs to a target server with TLS enabled, certificate authority (CA) bundles that a site administrator uploads using ghe-ssl-ca-certificate-install are not respected, and connections to the server fail.

When running ghe-config-apply, the process may stall with the message Deployment is running pending automatic promotion.

The mbind: Operation not permitted error in the /var/log/mysql/mysql.err file can be ignored. MySQL 8 does not gracefully handle when the CAP_SYS_NICE capability isn't required, and outputs an error instead of a warning.

After an administrator upgrades from GitHub Enterprise Server 3.7 or 3.8, to 3.9 or 3.10, MySQL may not start back up. For more information, see "Known issues with upgrades to your instance." [Updated: 2023-08-11]

After an administrator upgrades from GitHub Enterprise Server 3.7 or 3.8 to 3.9 or 3.10, I/O utilization will increase, and in some cases the instance's performance will be impacted. Reduced performance is due to the database server being upgraded from MySQL 5.7 to MySQL 8.0. For more information, see "Known issues with upgrades to your instance."

In rare circumstances, a small instance with both high availability and GitHub Actions configured may report that MSSQL replication is unhealthy after many upgrades with full upgrade packages. If you encounter this issue, contact GitHub Support.

On an instance in a cluster configuration with high availability configured, ghe-config-apply times out while waiting for hookshot-go to start on replica application nodes.

After an administrator enables maintenance mode from the instance's Management Console UI using Firefox, the administrator is redirected to the Settings page, but maintenance mode is not enabled. To work around this issue, use a different browser.

On an instance with a high-availability, geo-replication, or repository cache configuration, a known issue causes the SpokesRepairRepoReplicaJob and SpokesSyncCacheReplicaJob jobs to fail. This means repository replicas in cache servers are not updated, and will remain out of sync if the repository is updated. Additionally, if a regular repository replica becomes out of sync, repair attempts will fail, causing the corresponding repository network to be marked as failed until a network repair is triggered.

The network repair will eventually return the repository to a healthy state. However, this process can take several hours for very large and active repositories or networks, potentially leading to prolonged replication issues. [Updated: 2023-09-26]

When an administrator uses the -p flag with the ghe-support-bundle utility to collect data for a specific number of hours, the utility erroneously collects more logs than necessary. [Updated: 2023-10-13]

The settings for enabling scheduled reminders were added unintentionally to this release. Scheduled reminders are not officially supported. [Updated: 2023-10-17]

Jobs in a deprecated queue are not processed and may accumulate over time. These jobs are reflected in the monitor dashboard's "Aqueduct queue depth" graph as an increase in resource_activity. In some cases, a buildup of unprocessed jobs can result in memory exhaustion. If you observe memory exhaustion on your instance and see a high metric for resource_activity, contact GitHub Support. [Updated: 2023-10-26]

On an instance with GitHub Actions enabled, after an upgrade from GitHub Enterprise Server 3.8 or earlier, an internal exception could prevent successful completion of some operations, like upgrades or the configuration of new replica nodes for high availability. If this issue occurs, administrators may see the following error in /data/user/common/ghe-config.log.

Error occurred while executing servicing step 'Clone datatier login to secondary replica' for component CopyAvailabilityGroupSqlLogins during CopyAvailabilityGroupSqlLogins: Object reference not set to an instance of an object.

To resolve this issue, upgrade to the latest patch release of GitHub Enterprise Server. [Updated: 2023-12-04]

In some situations, large .adoc files stored in a repository do not render properly in the web UI. The raw contents are still available to view as plaintext. [Updated: 2023-10-31]

On an instance hosted in AWS, system time may lose synchronization with Amazon's servers after an administrator reboots the instance. [Updated 2023-11-10]

On an instance in a cluster configuration, restoration of a backup using ghe-restore will exit prematurely if Redis has not restarted properly. [Updated 2023-12-05]

On an instance with the HTTP X-Forwarded-For header configured for use behind a load balancer, all client IP addresses in the instance's audit log erroneously appear as 127.0.0.1. [Updated 2023-12-13]

During periods of high traffic, interruptions in service may occur due to insufficient resource allocations for internal components. [Updated 2024-01-23]

Redundant messages may cause an increase in the volume of events logged in /var/log/syslog. [Updated: 2024-03-08]

If a hotpatch upgrade requires the haproxy-frontend service to be restarted, the restart will hang if there are existing long-lived connections, such as browser web sockets or Git operations. No new connections will be accepted for up to 5 minutes. Any existing unfinished connections at this time will be disconnected. [Updated: 2024-06-17]

Instance administration

• To monitor the status of migrations in more detail, users with administrative SSH access to an instance can use the ghe-migrations utility to see the progress of individual migration groups. For more information, see "Command-line utilities."

• Site administrators can set a custom message for their users to see during a maintenance window. For more information, see "Enabling and scheduling maintenance mode."

• Site administrators can use the Manage GitHub Enterprise Server API to view and manage the maintenance status of an instance, including setting an IP exception list and modifying the message displayed to users during a maintenance window. For more information, see "REST API endpoints for managing GitHub Enterprise Server" in the REST API documentation.

• Site administrators can use the Manage GitHub Enterprise Server API to change the site admin password and to make changes to management console users. For more information, see "REST API endpoints for managing GitHub Enterprise Server" in the REST API documentation.

Authentication

• To help users access resources more securely, fine-grained personal access tokens are available in public beta. For more information, see "Managing your personal access tokens."

  • Users can create fine-grained personal access tokens with access to their personal repositories or, if permitted, organization-owned repositories.
  • Organization and enterprise owners can enable or disable the use of fine-grained personal access tokens in organization-owned repositories, and can use the REST API or GraphQL API to manage tokens in their organizations.
  • Users creating fine-grained tokens for an organization can add the pre-receive hooks permission to allow managing pre-receive hooks. For more information, see "Managing pre-receive hooks on your instance."

GitHub Advanced Security

• To find vulnerabilities in specific parts of a project, users with write access to a repository can filter code scanning alerts by language or by file path by using the search queries language: and path:. For more information, see "Assessing code scanning alerts for your repository."

• To help repository administrators and security managers quickly enable automatic code scanning without needing to configure a workflow, default setup for code scanning supports compiled languages including Go, Java, and C. Default setup is now available for all languages supported by CodeQL, except Swift. For more information, see "CodeQL code scanning for compiled languages" and Supported languages and frameworks in the CodeQL documentation.

• Repository administrators and security managers can choose which languages to include or exclude in default setup for code scanning. For more information, see "Configuring default setup for code scanning."

• To improve analysis of C# code, the release of CodeQL included with GitHub Enterprise Server 3.10 can scan projects that include features from C# 11. For more information, see What's new in C# 11 in the Microsoft documentation. Support for C# 11 is in beta and subject to change. CodeQL can scan projects built with C# 11 features, but does not analyse the code used for C# 11 features themselves.

• To help users find vulnerabilities in projects for Swift libraries and Apple apps, the release of CodeQL included with GitHub Enterprise Server 3.10 includes support for Swift, up to version 5.8.1, and Xcode, up to version 14.3.1. Support for Swift is in beta and subject to change. Swift analysis is not supported in default setup for code scanning, and requires the advanced setup. For more information, see "Configuring advanced setup for code scanning."

• To help identify steps to remediate leaked secrets, repository administrators and security managers can view metadata such as the secret owner, expiration date, and access rights for any active GitHub token leaked in a repository. This feature is in beta and subject to change. For more information, see "Managing alerts from secret scanning."

• Repository administrators, security managers, and organization and enterprise owners can view metrics for alerts generated by a specific custom pattern for secret scanning. This feature is in beta and subject to change. For more information, see "Defining custom patterns for secret scanning."

Dependabot

• Dependabot can automatically update the version of Node.js dependencies managed in the pnpm package manager. For more information, see "About Dependabot version updates."

• To avoid unnecessary compute cost, Dependabot updates are automatically paused in repositories where there has been no activity on pull requests created by Dependabot for 90 days. For more information about the criteria for Dependabot updates being paused, see "About Dependabot security updates" and "About Dependabot version updates."

• To avoid unnecessary compute cost, Dependabot stops automatically rebasing a pull request for version or security updates if the pull request has been open for 30 days.

Code security

• In the GitHub Advisory Database, users can search for any historical vulnerability recognized by the National Vulnerability Database. The "Unreviewed advisories" category has been backfilled to include vulnerabilities from previous years. For more information, see "Browsing security advisories in the GitHub Advisory Database."

• In the GitHub Advisory Database, users can search for malware advisories by using the query type:malware. Dependabot does not send alerts for malware advisories. For more information, see "Browsing security advisories in the GitHub Advisory Database."

• In the GitHub Advisory Database, users can search for advisories for the Hex package manager, including Elixir, Erlang, and more. Dependabot does not send alerts for Hex advisories. For more information, see "Browsing security advisories in the GitHub Advisory Database."

• Organization owners, security managers, and users with admin access to a repository can quickly enable or disable security features for a filtered selection of repositories from the "Security coverage" view in an organization's security overview. This feature is in beta and subject to change. For more information, see "Enabling security features for multiple repositories."

• Enterprise owners, organization owners, and security managers can quickly assess adoption of security features and exposure to security vulnerabilities across their enterprise. The enterprise-level "Security coverage" and "Security risk" views in security overview display data for repositories in each organization where the viewer is an organization owner or security manager. These views replace the "Overview" page in the "Code Security" tab for an enterprise. The risk metric for filtering the "Overview" page is no longer available. This feature is in beta and subject to change. For more information, see "About security overview."

• Users can find curated security advisories for the Swift ecosystem in the GitHub Advisory Database. For more information, see "About the GitHub Advisory database."

GitHub Actions

• For self-hosted GitHub Actions runners on this GitHub Enterprise Server release, the minimum required version of the GitHub Actions Runner application is 2.304.0. See the release notes for this version in the actions/runner repository. If your instance uses ephemeral self-hosted runners and you've disabled automatic updates, you must upgrade your runners to this version of the Runner application before upgrading your instance to this GitHub Enterprise Server release. [Updated: 2024-04-25]

• Organization owners can increase instance security by preventing members from creating self-hosted runners at the repository level. For more information, see "Disabling or limiting GitHub Actions for your organization."

• Users with admin access to a repository can allow external systems and third-party services to approve or reject deployments across organizations, repositories, and environments by enabling custom deployment protection rules. This feature is in beta and subject to change. For more information, see "Managing environments for deployment."

• The option to execute custom scripts on a self-hosted runner is no longer is beta. For more information, see "Running scripts before or after a job."

• To prevent unnecessary transfer of OIDC tokens between workflows, to fetch an OIDC token generated within a reusable workflow that is outside their enterprise or organization, users must set the id-token permission to write in the workflow or specific job where the reusable workflow is called. For more information, see "Configuring OpenID Connect in cloud providers."

• Repository administrators, organization owners, and users with the manage_runners:enterprise scope for enterprises can use the REST API to create ephemeral, just-in-time (JIT) runners that can perform at most one job before being automatically removed from the repository, organization, or enterprise. For more information, see "Security hardening for GitHub Actions."

Community experience

• To improve the accuracy of marked answers in discussions, and reduce the burden on users to duplicate their text to get their answer marked as correct, users can mark threaded replies as the answer to a question.

• To improve content organization and topic discoverability, GitHub Discussions maintainers can group discussion categories into sections.

Repositories

• To prevent unnecessary repository removal, the API for managing the repositories accessible by a GitHub App in your organization has been updated to fail early if the application is currently granted access to all repositories in the organization. This API can only be used to remove a repository when the application has been granted access to an explicit list of repositories. For more information, see "REST API endpoints for GitHub App installations."

• Repository administrators can ensure the security and stability of branches by requiring pull request approval by someone other than the last pusher. For more information, see "About protected branches."

Projects

• Projects is no longer in public beta, and is now considered generally available. For more information, see "About Projects."

• To control the amount of work in progress and promote focus, on a board layout, users with admin access to a project can set a recommended limit on the number of items in a column. For more information, see "Customizing the board layout."

• To determine the default access rights organization members have to projects where they haven't been granted individual access, organization owners can set a base role for projects. For more information, see "Managing access to your projects."

• To share a pre-configured project with other people in an organization, users with admin access to a project can set the project as a template. This feature is in beta and subject to change. For more information, see "Managing project templates in your organization."

• In a table layout, users can select and update multiple cells at once by clicking and dragging or using the Shift or Ctrl/Command key.

Commits

• When editing a file in the user interface, users with permission to bypass branch protection rules receive a note if their commit will bypass a rule, with the option to create a new branch instead of committing directly to the protected branch. Previously, the commit was added to the protected branch directly, without indication that a rule was being bypassed.

• When using git push from the command line, users with permission to bypass branch protection rules receive a note if they have pushed a commit that bypasses a rule. Previously there was no indication after a Git push that branch rules had been bypassed.

Markdown

• Users can include mathematical expressions within Markdown by using LaTeX syntax delimited by $ characters and backticks. For more information, see "Writing mathematical expressions."

Accessibility

• To make GitHub inclusive to all developers, GitHub has improved color contrast of the default light and dark themes, making them accessible to all users. These changes were made to Primer, GitHub's Design System. For more information, see GitHub Accessibility.

Field names and destinations for some service logs on GitHub Enterprise Server have changed in this release and the prior release. If any tooling or processes in your environment rely on specific field names within logs, or log entries in specific files, the following changes may affect you.

• level is now SeverityText.
• log_message, msg, or message is now Body.
• now is now Timestamp.
• Custom field names such as gh.repo.id or graphql.operation.name use semantic names.
• Log statements that the instance would previously write to auth.log, ldap.log, or ldap-sync.log now appear in containerized logs for github-unicorn if the statement originated from a web request, or in logs for github-resqued if the statement originated from a background job. For more information about containerized logs, see "About system logs."

For a full list of field mappings, download the OpenTelemetry attribute mapping CSV for GitHub Enterprise Server 3.9 and the OpenTelemetry attribute mapping CSV for GitHub Enterprise Server 3.10. This change is part of GitHub's gradual migration to internal semantic conventions for OpenTelemetry, and additional field names will change in upcoming releases.

Users who use pull requests with protected branches may be affected by the following security measures.

• Merge commits created locally and pushed to a protected branch are rejected if the contents of the commit differ from the merge commit predicted by GitHub.
• If the branch protection rule for dismissing stale reviews is active, an approving review is dismissed if the merge base changes after the review was submitted. The merge base is the commit that is the latest common ancestor of the pull request branch and the protected branch.
• A pull request approval only counts towards the pull request it was submitted for. Previously, approvals were gathered across multiple independent pull requests if the pull request branches pointed to the same commit and targeted the same base branch.

The PUT and DELETE operations on the /installations/{installation_id}/repositories/{repository_id} endpoint are no longer functional for the management of GitHub App installations. You can add or remove a repository from an app installation using the documented APIs instead. For more information, see "REST API endpoints for GitHub App installations."

On an instance with a GitHub Advanced Security license, to make it easier to assess vulnerabilities to exposed secrets, enterprise owners and organization owners receive a single email with the results of the historical scan for secrets that is performed when secret scanning is first enabled in an organization or enterprise. Previously, secret scanning sent an email for each repository where secrets were detected. For more information, see "About secret scanning."

On an instance with a GitHub Advanced Security license, in the "Files changed" view of pull requests, GitHub only displays code scanning alerts for vulnerabilities detected in lines that a pull request has changed. Previously, code scanning displayed all alerts unique to the pull request branch, even if they were unrelated to the changes the pull request introduced.

To generate backups of an instance more quickly, in GitHub Enterprise Server Backup Utilities 3.10.0 and later, the job for pruning snapshots is no longer performed as part of the ghe-backup tool. Site administrators can prune snapshots manually or on a schedule by running the ghe-prune-snapshots job. For more information, see Scheduling backups in the github/backup-utils repository.

To reduce the data transfer required to regularly back up an instance, GitHub Enterprise Server Backup Utilities 3.10.0 and later allows administrators to perform incremental backups of a MySQL 8 database. For more information, see Making a Differential or Incremental Backup in the MySQL documentation.

After an administrator upgrades from GitHub Enterprise Server 3.7 or 3.8 to 3.9 or 3.10, I/O utilization will increase, and in some cases the instance's performance will be impacted. Reduced performance is due to the database server being upgraded from MySQL 5.7 to MySQL 8.0. For more information, see "Known issues with upgrades to your instance."

After an administrator upgrades from GitHub Enterprise Server 3.7 or 3.8, to 3.9 or 3.10, MySQL may not start back up. For more information, see "Known issues with upgrades to your instance." [Updated: 2023-08-11]

In GitHub Enterprise Server 3.10 and later, the requirements for TLS security levels have changed due to an upgrade to containers in the underlying OS. On an instance with GitHub Actions enabled and a custom TLS certificate, users may experience disruptions with workflow runs if the TLS certificate uses weak encryption. Workflow runs will not trigger, and the following error message will appear in system logs for babeld.

CA certificate key too weak

To resolve this issue, confirm that your certificate complies with level 2 of the OpenSSL security specification. For more information, see SSL_CTX_set_security_level in the OpenSSL docs. For more information about reviewing your instance's logs, see "About system logs".

If the error appears in babeld logs because your TLS certificate does not comply with level 2 of the specification, you must create and upload a new certificate with stronger security before you upgrade to GitHub Enterprise Server 3.10 or later. For more information, see "Configuring TLS." [Updated: 2023-10-26]

When enabling CodeQL via default setup at scale, some checks related to GitHub Actions are omitted, potentially preventing the process from completing.

On an instance in a cluster configuration, after you upgrade nodes other than the primary MySQL node and before you upgrade the primary MySQL node, the following output may appear multiple times after you run ghe-config-apply.

Error response from daemon: conflict: unable to delete IMAGE_ID (cannot be forced) - image is being used by running container CONTAINER_ID

You can safely ignore this message.

Custom firewall rules are removed during the upgrade process.

On an instance in a high-availability configuration, passive replica nodes accept Git client requests and forward the requests to the primary node.

The mbind: Operation not permitted error in the /var/log/mysql/mysql.err file can be ignored. MySQL 8 does not gracefully handle when the CAP_SYS_NICE capability isn't required, and outputs an error instead of a warning.

When using an outbound web proxy server, the ghe-btop command may fail in some circumstances with the error "Error querying allocation: Unexpected response code: 401".

If an instance is configured to forward logs to a target server with TLS enabled, certificate authority (CA) bundles that a site administrator uploads using ghe-ssl-ca-certificate-install are not respected, and connections to the server fail.

When running ghe-config-apply, the process may stall with the message Deployment is running pending automatic promotion.

During the validation phase of a configuration run, a No such object error may occur for the Notebook and Viewscreen services. This error can be ignored as the services should still correctly start.

If the root site administrator is locked out of the Management Console after failed login attempts, the account will not unlock automatically after the defined lockout time. Someone with administrative SSH access to the instance must unlock the account using the administrative shell. For more information, see "Troubleshooting access to the Management Console."

In rare circumstances, a small instance with both high availability and GitHub Actions configured may report that MSSQL replication is unhealthy after many upgrades with full upgrade packages. If you encounter this issue, contact GitHub Support. [Updated: 2023-09-04]

After an administrator enables maintenance mode from the instance's Management Console UI using Firefox, the administrator is redirected to the Settings page, but maintenance mode is not enabled. To work around this issue, use a different browser. [Updated: 2023-09-19]

On an instance in a cluster configuration with high availability configured, ghe-config-apply times out while waiting for hookshot-go to start on replica application nodes. [Updated: 2023-09-21]

On an instance with a high-availability, geo-replication, or repository cache configuration, a known issue causes the SpokesRepairRepoReplicaJob and SpokesSyncCacheReplicaJob jobs to fail. This means repository replicas in cache servers are not updated, and will remain out of sync if the repository is updated. Additionally, if a regular repository replica becomes out of sync, repair attempts will fail, causing the corresponding repository network to be marked as failed until a network repair is triggered.

The network repair will eventually return the repository to a healthy state. However, this process can take several hours for very large and active repositories or networks, potentially leading to prolonged replication issues. [Updated: 2023-09-26]

When an administrator uses the -p flag with the ghe-support-bundle utility to collect data for a specific number of hours, the utility erroneously collects more logs than necessary. [Updated: 2023-10-13]

The settings for enabling scheduled reminders were added unintentionally to this release. Scheduled reminders are not officially supported. [Updated: 2023-10-17]

Jobs in a deprecated queue are not processed and may accumulate over time. These jobs are reflected in the monitor dashboard's "Aqueduct queue depth" graph as an increase in resource_activity. In some cases, a buildup of unprocessed jobs can result in memory exhaustion. If you observe memory exhaustion on your instance and see a high metric for resource_activity, contact GitHub Support. [Updated: 2023-10-26]

On an instance with GitHub Actions enabled, after an upgrade from GitHub Enterprise Server 3.8 or earlier, an internal exception could prevent successful completion of some operations, like upgrades or the configuration of new replica nodes for high availability. If this issue occurs, administrators may see the following error in /data/user/common/ghe-config.log.

Error occurred while executing servicing step 'Clone datatier login to secondary replica' for component CopyAvailabilityGroupSqlLogins during CopyAvailabilityGroupSqlLogins: Object reference not set to an instance of an object.

To resolve this issue, upgrade to the latest patch release of GitHub Enterprise Server. [Updated: 2023-12-04]

In some situations, large .adoc files stored in a repository do not render properly in the web UI. The raw contents are still available to view as plaintext. [Updated: 2023-10-31]

On an instance hosted in AWS, system time may lose synchronization with Amazon's servers after an administrator reboots the instance. [Updated 2023-11-10]

On an instance in a cluster configuration, restoration of a backup using ghe-restore will exit prematurely if Redis has not restarted properly. [Updated 2023-12-05]

On an instance with the HTTP X-Forwarded-For header configured for use behind a load balancer, all client IP addresses in the instance's audit log erroneously appear as 127.0.0.1. [Updated 2023-12-13]

During periods of high traffic, interruptions in service may occur due to insufficient resource allocations for internal components. [Updated 2024-01-23]

Redundant messages may cause an increase in the volume of events logged in /var/log/syslog. [Updated: 2024-03-08]

If a hotpatch upgrade requires the haproxy-frontend service to be restarted, the restart will hang if there are existing long-lived connections, such as browser web sockets or Git operations. No new connections will be accepted for up to 5 minutes. Any existing unfinished connections at this time will be disconnected. [Updated: 2024-06-17]

Upcoming deprecation of team discussions

• GitHub will deprecate team discussions for users in GitHub Enterprise Server 3.13. In GitHub Enterprise Server 3.10, a banner appears atop teams' discussions with information about the deprecation, including a link to tooling to migrate existing team discussions to GitHub Discussions. For more information, see "About team discussions" and "About discussions." [Updated: 2024-03-04]