Question: Investigate solr performance, diagnose bottlenecks and develop optimizations?

@luddaniel first, I'd like to point you toward a SQL query that might help as a workaround. It's from the useful queries doc linked from URL

List the titles of datasets without files (4.20):
select dvo.createdate as date_of_creation, dfv.value as title, dvo.publicationdate as publication_date
from datasetfieldvalue dfv
join datasetfield df on dfv.datasetfield_id = df.id
join datasetversion dv on df.datasetversion_id = dv.id
join dvobject dvo on dv.dataset_id = dvo.id
where datasetfield_id not in
(select id
from datasetfield
where datasetversion_id in
(select id
from datasetversion
where id in
(select datasetversion_id
from filemetadata)))
and datasetfield_id in
(select id
from datasetfield
where datasetfieldtype_id in
(select id
from datasetfieldtype
where name = 'title'))
order by dvo.createdate;

I see what you mean about fileCount. If I search for my dataset at URL I see it like this:

"fileCount": 111,

However, this number isn't indexed into Solr. It gets pulled from the database like this:

nullSafeJsonBuilder.add("fileCount", dv.getFileMetadatas().size());

It looks like I added it in 055fa5f as part of this PR:

- #6623

@landreev even asked why we don't just index the number. I guess we should have. 😞

Investigate solr performance, diagnose bottlenecks and develop optimizations
This primarily concerns busy instances with large amounts of indexed metadata. Such as the IQSS prod. installation. There's some evidence of Solr becoming the performance bottleneck during high service load. Facet queries are especially suspect. But this has the potential to benefit any busy instance, hence the issue in the main repo.

@kcondon got the ball rolling on this and has already found and applied some useful info. He has started a google doc documenting his work. We may link it here and we will try to document the most useful things we find here (and in the guides, eventually). Kevin's out today, but we had a conversation and decided I'll go ahead an open the issue to make sure this work is scheduled and tracked; striking it while it's hot, etc.

Spike issue to investigate Solr performance issues and identify specific actions to take. An issue will be created for each action that is identified. (As per 2023/07/30 prioritization meeting).

There's another low hanging fruit-like improvement that can be handled as a compact, self-contained issue: Inside SearchIncludeFragment (the class file that runs the search for the collection page - i.e., where most searches issued by Dataverse originate) every search is literally run twice, in order to obtain the result counts for the object types NOT currently selected by the user. While the second search is necessary in order to obtain these count(s), it can be run in a more economical way, that can reduce the amount of work for solr to perform by up to 50%.
The details are spelled out in a #dv-tech thread and can be copy-and-pasted into an issue once we open it.

@luddaniel I just looked at #8941. If it can be addressed by simply indexing fileCount as another numeric field during indexing, that does sound very straightforward, and I don't know why we haven't implemented it yet/have overlooked it.
Generally, we absolutely need to reduce the number of SQL lookups that are required when we process list of results as they come from Solr. Hopefully we can address it soon.

We had a series of prod. auto-restarts again yesterday. This is definitely caused by solr stopping responding, this is confirmed now.
A bandaid, but hopefully an effective temp. workaround was applied on the solr side, setting up "circuit breaker" config options. This makes solr start dropping new connections with 503 when certain threshold of memory and/or cpu utilization is reached, rather than hanging. (the observed search page freezes were accompanied by the cpu utilization approaching 800% - on the 8-cpu node that is dvn-cloud-solr). Thanks/kudos to @kcondon for researching/figuring out the above.

Also noticed that clicking on basic search with no parameters spikes solr cpu. Clicking multiple times raises cpu use rapidly to where I can simulate the 700+% cpu load on solr. Refreshing the homepage or clicking on facets does not do this beyond an initial spike, then levels off. I was able to trigger the cpu circuit breaker, I think, using this method.

I'm implementing the following application side improvements, and hoping to make a PR soon. (more may be added to the list)
- Adding app-side support for the Solr-size circuit breakers. Enabling those makes Solr start dropping requests with 503s (instead of dying) under heavy load. As currently implemented however, the page does not detect that condition and doesn't handle it in any special way, and simply shows zero search results/empty collection page. It needs to show something intelligent, along the lines of "search engine is experiencing heavy load, please try again later".
- Optimizing the second search that the page repeats solely for the purpose of populating the type counts facets for the _unchecked_ types. There is absolutely no need to repeat the **same exact search**, with all its super expensive parts, like all the other facets. There's no need to search for anything other than these one or two total counts; or to even run this second search, if all 3 type facets happen to be checked.
- Adding `nofollow` to all the facet links and such, that we don't want crawlers/bots to touch. This will obviously only help with the nicer bots that take hints and follow rules, but better than nothing.
- An off switch for facets support on the dataverse page. Again, facets are super expensive; it should be far better to be able to display the page with just the search cards and "facets are temporarily unavailable because of load issues" in the left column, than to have an empty page/have a "solr down" error page/have the page load take an hour.

fileCount in Solr Documents and API performance
Hello, I have recently worked on a customized tool that use Dataverse Installation Search API and I was questionning myself on some choices done in code.

My need was to find all datasets without one custom field and without any files.
Customized field is ok `/api/v1/search?q=-alternativeURL:*&type=dataset`
But no files... I found that json reponse had fileCount property constructed by java code after solr result.
Problem is : search is really slow if you ask lot of data.
Ex : `/api/v1/search?q=*&type=dataset&subtree=root&sort=name&order=asc&per_page=1000` takes 37 seconds for a 2.3Mo json file of 1000 datasets on our Dataverse installation (1991 datasets and 29844 datafiles).
I ended up doing it using mostly SQL queries.

1) I think it would be intesting to have information on solr document instead, in order to have better performance and mostly be able to ask `fileCount:0` in Dataverse Repository UI search bar or using Search API.
Or maybe display it on Dataverse Installation search snippet result or in a customized tool in my case (ex: 66 files).
I have seen that the question and the idea had exist in the past during `fileCount` implementation : #6601 #6623

2) I'm a bit disturbed to see simple query based on solr search hiting 37 seconds for only 1000 elements.
My guess is there is too much SQL done afterward SOLR result.
The interest of using SOLR is to have ultra fast response, my humble opinion is that it's start to lose interest, it would be faster using 100% SQL or 100% Solr don't you think ?
As Java developper I do understand that complex application make those choices really complicated due to maintenance of SQL queries and critetrias or to not overload SOLR documents.
But there is a performance danger to make SQL queries for each documents of SOLR found.
I might be off topic, feel free to correct me.

I can't wait to read you and, as always, thanks for you time and services for the community of Dataverse software.

@luddaniel sorry for the slow response. I'm trying to understand what you're up to. 😄

Does the field have to be custom? Are you simply saying, "I want to know which datasets are not using field x?"

(By "no files" I assume you mean "no documents". Solr calls results "documents".)

If you're concerned about performance, please try querying Solr directly to see if it's faster. The Dataverse Search API does (sadly) add some overhead but I'm not sure how much.

If the SQL queries are working for you perhaps you can make a pull request to make an API endpoint.

What is this tool you've built? 😄 You've piqued my curiosity. 😄

Maybe we can chat in real time at URL some day. You're welcome to join!

Hello @pdurbin :) Thank you for answering

By "no files" I mean "To retrieve Dataset without Datafile".
Maybe my text was mixing too much informations ^^

In other words, I want a SOLR field that tells me this dataset has 32 files. So I can ask Dataverse Search with : `fileCount=0` so I retrieve datasets without files or `fileCount>10` so I retrieve datasets with at least 10 files.

That's what I want to do but I cannot due to lack of 1 SOLR field that could contains informations on datafiles owned by a dataset (1 dataset = 1 solr document)
Ex : fileCount (int type) or fileIDs (array type)

Sadly, I don't have enough time at work to contribute at the moment but our team is thinking about contributing to Dataverse software, someday we will push code for sure !

We created a little python web application to help our support team (non technical persons) of our Dataverse Installation.
It uses Dataverse API Endpoints, SQL queries, Dataverse Search API. At this moment :

1. Shows all datasets without datafile and with empty field alternativeUrl (so we can ask scientists to populate their dataset with data)
2. Import list of email to add them in group (massive user configuration)
3. Help to merge non admin users by selecting in 2 lists of users one to merge and one user to keep.

I will sign in, hope to chat with you soon. It could be easier to exchange using screensharing.
Genuinely, Ludovic.


Question: APM data shows telemetry being called ~30 times a day?

Another idea that would improve our visibility of telemetry is URL What do you think?

APM data shows telemetry being called ~30 times a day
We recently enabled APM on the `overview-aws-eu-west-1` kibana (QA environment used to monitor all the other QA deployments).
While analyzing the data, I noticed something interesting and after talking to @Bamieh I wanted to open an issue to track this.

This cluster is running 7.16.0 (meaning URL is already merged into it) and it seems like it's calling the telemetry API way more often than one would expect.

The latency of each call is `10-60s` and they happen once about every hour.
Some contain failures.

APM data from past 7 days
Discover telemetry call transactions,refreshInterval:(pause:!t,value:0),time:(from:now-7d%2Fd,to:now))&_a=(columns:!(transaction.name,observer.hostname,client.ip,host.ip,client.geo.location,client.geo.country_name,labels.deploymentName),filters:!(('$state':(store:appState),meta:(alias:!n,disabled:!f,index:apm_static_index_pattern_id,key:processor.name,negate:!f,params:(query:transaction),type:phrase),query:(match_phrase:(processor.name:transaction))),('$state':(store:appState),meta:(alias:!n,disabled:!f,index:apm_static_index_pattern_id,key:service.name,negate:!f,params:(query:kibana),type:phrase),query:(match_phrase:(service.name:kibana))),('$state':(store:appState),meta:(alias:!n,disabled:!f,index:apm_static_index_pattern_id,key:transaction.name,negate:!f,params:!('POST%20%2Fapi%2Ftelemetry%2Fv2%2Fclusters%2F_stats'),type:phrases),query:(bool:(minimum_should_match:1,should:!((match_phrase:(transaction.name:'POST%20%2Fapi%2Ftelemetry%2Fv2%2Fclusters%2F_stats')))))),('$state':(store:appState),meta:(alias:!n,disabled:!f,index:apm_static_index_pattern_id,key:labels.deploymentName,negate:!f,params:(query:overview-aws-eu-west-1),type:phrase),query:(match_phrase:(labels.deploymentName:overview-aws-eu-west-1)))),grid:(),hideChart:!f,index:apm_static_index_pattern_id,interval:auto,query:(language:kuery,query:''),sort:!(!('@timestamp',desc))))
One of the errors found in the telemetry request

The repetition could be caused by URL And the fix will ship in `8.0.0`.

In the meanwhile, I'd recommend upgrading to `7.16.2` to make use of the caching mechanism so it will save many ES requests.

After looking at the APM data, I also identified 2 potential improvements so we can reduce the amount of time it takes to generate the telemetry report:
- URL
- URL

I also added them to our Meta issue for Telemetry performance improvements URL

@lizozom I was going to ask the same since we covered most of it. I'll go ahead and close it. Feel free to reopen if needed.

[Meta][Telemetry] Reduce telemetry footprint
Reducing the telemetry footprint ensures future scalability of our telemetry and puts a system in place to enable performance optimizations for our collection methods. This also helps reduce the number of people opting out of telemetry in cases where telemetry is causing significant spikes in resources.

- [x] Exponentially backs off retry attempts for sending usage data #115221
- [x] Cache stats to avoid #117084
- [x] [Telemetry] Use lastReported timestamp on the 'browser' as well #87846
- [x] Performance improvements on collectors:
 - [x] URL
 - [x] URL
- [x] URL
- [x] #123837
- [x] Add telemetry around the time it is taking for grabbing the telemetry stats #119468
- [x] #122516
- [x] #122572
- [x] Audit Usage collection with high Kibana/ES performance cost for some clusters #117489
- [ ] Add load-test suites to identify our performance limitations
 - [x] Concurrent users URL
 - [ ] #122637
- [ ] #93770
- [x] Lens using a separate UI collection #48352
- [ ] URL
- [ ] #135922

Should we also tackle URL IMHO, it will reduce the number of requests to only 1 every 24h (no matter how many users the cluster has).


Question: Question for the target label?

The issue was actually caused by a boundary being rendered. This is resolved in 1.0.9.

FYI, you might find it more convenient to use the `task.get_demos()` functionality, rather than parsing the demo folders yourself (see here). If you pass in the live=False flag, then it gives you a list of demos loaded from file. You can also set `image_paths=True` if you want the paths rather than the images themselves.

Question for the target label
Thank you for your excellent work!

I noticed that in the `taskvars_target_label_zrange.json` file, you provide the key actions and the corresponding object labels for each task. For example,

"close_grill+0": [
 {
 "action": "grasp",
 "object": {
 "name": "grill handle",
 "fine": [87],
 "coarse": [87, 86]
 }
 },
 {
 "action": "move grasped object up"
 }
 ].

I am curious about the meaning of the numbers in `"fine": [87]` and `"coarse": [87, 86]`. What do these values represent, and where do they come from?

Looking forward to your response！

The RLBench simulator provides groundtruth segmentation masks. These numbers are the instance ids of the object in the groundtruth mask. We manually annotated the labels.

Sorry, I have another confusion. I referred to a solution mentioned in an issue from RLBench: URL and I tried to obtain the instance IDs of the object in task `take_shoes_out_of_box`. However, the result I generated seems to not match the values in the script you provided.

I obtained a total of 37 object IDs, and the ID for the box lid is 35,

Image

but in the script you provided, it is 96.

"take_shoes_out_of_box+0": [
 {
 "action": "grasp",
 "object": {
 "name": "box lid",
 "fine": [96],
 "coarse": [92, 96]
 }
 },
 {
 "action": "move grasped object up"
 },
 {
 "action": "grasp",
 "object": {
 "name": "shoe",
 "fine": [82],
 "coarse": [82]
 }
 },
 {
 "action": "move grasped object out"
 },
 {
 "action": "grasp",
 "object": {
 "name": "shoe",
 "fine": [102],
 "coarse": [102]
 }
 },
 {
 "action": "move grasped object out"
 }
 ]

I would like to ask again, how are these instance IDs obtained?

Looking forward to your reply and would greatly appreciate it!

Please follow the instruction in our dataset generation to get the groundtruth segmentation masks. Our annotated ids are related to these masks.

@cshizhe @chenzixuan99 I have the same problem as you, can you provide the code to get the instance ID? I want to know the labels of different objects. I will be very grateful.

segmentation mask rendering problem?
Hi,

I was trying to generate a bunch of images and masks for the place_cups task. I generated the dataset using the following command:

python tools/dataset_generator.py --save_path ./debug --tasks place_cups --image_size '256,256' --episodes_per_task 1 --variations 1

I assume each number in the segmentation map corresponds to certain types of objects. And indeed, some of the numbers are correctly associated with the object. For example, the image looks like:

0022

The number 36 corresponds to:

36_22

which seems to be correct.

However, I found that some number corresponds to the wrong region. Specifically, number 35 corresponds to:

35_22

---

I guess this is not an expected phenomenon? Is this a bug of mask rendering? Or how should I get the correct mask for objects?

Thanks a lot!

Hi @HaozhiQi.

This is not expected behaviour, so theres probably a bug in the mask decoding. I'll take a look. Thanks for pointing this out.
Would it be possible for you send me a reproducible script that shows the bug?

Hi @stepjam ,

Sure! I use RLBench with version 1.0.9 (
git checkout 1.0.9
) and installed with
python setup.py install --user
. My pyrep version is 1.2.

As I mentioned above, I generated the observations using the official script:

python tools/dataset_generator.py --save_path ./debug --tasks place_cups --image_size '256,256' --episodes_per_task 1 --variations 1

Then I run the following script:

import os
import cv2
import shutil
import numpy as np

from tqdm import tqdm
from glob import glob

def main():
 src_dir = './debug/place_cups/'
 tar_dir = './debug/parsed_place_cups/'

 variations = sorted(glob(f'{src_dir}/variation*/'))
 for v_i, variation in enumerate(tqdm(variations)):
 episodes = sorted(glob(f'{variation}/episodes/episode*/'))
 for e_i, episode in enumerate(episodes):
 image_path = f'{episode}/front_rgb/'
 images = glob(f'{image_path}/*.png')
 os.makedirs(f'{tar_dir}/image/v{v_i:03d}_e{e_i:03d}/', exist_ok=True)
 for image in images:
 im_id = int(image.split('/')[-1].split('.')[0])
 tar_path = f'{tar_dir}/image/v{v_i:03d}_e{e_i:03d}/{im_id:04d}.png'
 shutil.copy2(image, tar_path)
 mask = cv2.imread(image.replace('rgb', 'mask'), 0)
 obj_ids = list(np.unique(mask))
 for o_id, obj_id in enumerate(obj_ids):
 im = cv2.imread(image)
 masked_image = (mask == obj_id)[..., None] * im
 os.makedirs(f'{tar_dir}/{v_i}_{e_i}/', exist_ok=True)
 cv2.imwrite(f'{tar_dir}/{v_i}_{e_i}/{obj_id}_{im_id}.png', masked_image)

if __name__ == '__main__':
 main()

Hi @stepjam,

I observed something weird for the place cups task. For id 33, it gives:
33_214
There is a small green segment on top of the cup, which should not be the case.

Moreover, when I'm playing with other tasks, a similar issue happened. The problematic task is block_pyramid. Using a similar script (see the bottom), it produces the following results:

The id 53 corresponds to:
53_97
which should be a green region indicating the target.

Moreover, I though each id will correspond to a single object, but id 54 and 55 give:
54_817
55_20

This seems to be a systematic error rather than a bug for a single task. I'm thinking there may exist some bug in the .ttm file generation? Could you take a look at it?

The script I used to parse block_pyramid is:

python tools/dataset_generator.py --save_path ./debug --tasks block_pyramid --image_size '256,256' --episodes_per_task 1 --variations 1

import os
import cv2
import shutil
import numpy as np

from tqdm import tqdm
from glob import glob

def main():
 src_dir = './debug/block_pyramid/'
 tar_dir = './debug/parsed_block_pyramid/'

 variations = sorted(glob(f'{src_dir}/variation*/'))
 for v_i, variation in enumerate(tqdm(variations)):
 episodes = sorted(glob(f'{variation}/episodes/episode*/'))
 for e_i, episode in enumerate(episodes):
 image_path = f'{episode}/front_rgb/'
 images = glob(f'{image_path}/*.png')
 os.makedirs(f'{tar_dir}/image/v{v_i:03d}_e{e_i:03d}/', exist_ok=True)
 for image in images:
 im_id = int(image.split('/')[-1].split('.')[0])
 tar_path = f'{tar_dir}/image/v{v_i:03d}_e{e_i:03d}/{im_id:04d}.png'
 shutil.copy2(image, tar_path)
 mask = cv2.imread(image.replace('rgb', 'mask'), 0)
 obj_ids = list(np.unique(mask))
 for o_id, obj_id in enumerate(obj_ids):
 im = cv2.imread(image)
 masked_image = (mask == obj_id)[..., None] * im
 os.makedirs(f'{tar_dir}/{v_i}_{e_i}/', exist_ok=True)
 cv2.imwrite(f'{tar_dir}/{v_i}_{e_i}/{obj_id}_{im_id}.png', masked_image)

if __name__ == '__main__':
 main()

Hi,
There may be a bug. Could you first swap to loading demos through the Environment interface? In your script that you linked, you are nor decoding the masks correctly.

Hi,

Sorry, I didn't realize the mask is converted to 3 channel RGB image in the tools/dataset_generator.py so my parsing mask code is wrong. After changing it to single channel mask image, it works as expected.

Thanks a lot for your time!


Question: multiprocessing: serialization must ensure that contexts are compatible (the same)?

Hmm, interesting. I thought that the auto-merge feature would also auto-close the issue, but it didn't.

multiprocessing: serialization must ensure that contexts are compatible (the same)
BPO | 33196
--- | :---
Nosy | @pitrou, @vstinner, @taleinat, @applio, @arcivanov, @augustogoulart
Files | test_lock_sigsegv.py testing_on_fedora.png coredump: coredump (Fedora 29)

 *Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.*

 Show more details

GitHub fields:

python
assignee = None
closed_at = None
created_at =
labels = ['3.8', '3.7', 'library', 'type-crash']
title = 'multiprocessing: serialization must ensure that contexts are compatible (the same)'
updated_at =
user = ' URL

bugs.python.org fields:

python
activity =
actor = 'davin'
assignee = 'none'
closed = False
closed_date = None
closer = None
components = ['Library (Lib)']
creation =
creator = 'arcivanov'
dependencies = []
files = ['47510', '47931', '47933']
hgrepos = []
issue_num = 33196
keywords = []
message_count = 11.0
messages = ['314762', '314792', '329491', '329719', '329845', '329892', '329893', '329898', '329908', '329909', '329913']
nosy_count = 6.0
nosy_names = ['pitrou', 'vstinner', 'taleinat', 'davin', 'arcivanov', 'augustogoulart']
pr_nums = []
priority = 'normal'
resolution = None
stage = 'needs patch'
status = 'open'
superseder = None
type = 'crash'
url = ' URL
versions = ['Python 3.6', 'Python 3.7', 'Python 3.8']

### Linked PRs
* gh-107275
* gh-108377
* gh-108378

cc @vstinner I sent a PR for this, are you the right person to review it?

@albanD:

I'm not available to review your PR. Maybe @pitrou can review it.

Patch apparently caused regression for nested multiprocessing calls in 11.5. #108520

Nested multiprocessing leads to `AttributeError: is_fork_ctx` with `forkserver` or `spawn` methods
# Bug report

### Checklist

- [X] I am confident this is a bug in CPython, not a bug in a third-party project
- [X] I have searched the CPython issue tracker,
and am confident this bug has not been reported before

### CPython versions tested on:

3.11

### Operating systems tested on:

Linux

### Output from running 'python -VV' on the command line:

Python 3.11.5 (main, Aug 26 2023, 00:26:34) [GCC 12.2.1 20220924]

### A clear and concise description of the bug:

Using nested `multiprocessing` (i.e. spawn a child process inside a child process) is broken as of Python 3.11.5, leading to an attribute error.

python
Process Process-1:
Traceback (most recent call last):
 File "/usr/local/lib/python3.11/multiprocessing/process.py", line 314, in _bootstrap
 self.run()
 File "/usr/local/lib/python3.11/multiprocessing/process.py", line 108, in run
 self._target(*self._args, **self._kwargs)
 File "/io/pyi_multiprocessing_nested_process.py", line 15, in process_function
 process.start()
 File "/usr/local/lib/python3.11/multiprocessing/process.py", line 121, in start
 self._popen = self._Popen(self)
 ^^^^^^^^^^^^^^^^^
 File "/usr/local/lib/python3.11/multiprocessing/context.py", line 224, in _Popen
 return _default_context.get_context().Process._Popen(process_obj)
 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
 File "/usr/local/lib/python3.11/multiprocessing/context.py", line 288, in _Popen
 return Popen(process_obj)
 ^^^^^^^^^^^^^^^^^^
 File "/usr/local/lib/python3.11/multiprocessing/popen_spawn_posix.py", line 32, in __init__
 super().__init__(process_obj)
 File "/usr/local/lib/python3.11/multiprocessing/popen_fork.py", line 19, in __init__
 self._launch(process_obj)
 File "/usr/local/lib/python3.11/multiprocessing/popen_spawn_posix.py", line 47, in _launch
 reduction.dump(process_obj, fp)
 File "/usr/local/lib/python3.11/multiprocessing/reduction.py", line 60, in dump
 ForkingPickler(file, protocol).dump(obj)
 File "/usr/local/lib/python3.11/multiprocessing/synchronize.py", line 106, in __getstate__
 if self.is_fork_ctx:
 ^^^^^^^^^^^^^^^^
AttributeError: 'Lock' object has no attribute 'is_fork_ctx'
Results: [1]

Minimal code example below. Invoke with argument `fork`, `forkserver` or `spawn`. `fork` will work. `forkserver` and `spawn` will both raise the above error. All three variants work with Python 3.11.4.

python
import sys
import multiprocessing

def nested_process_function(queue):
 print("Running nested sub-process!")
 queue.put(2)

def process_function(queue):
 print("Running sub-process!")
 queue.put(1)

 process = multiprocessing.Process(target=nested_process_function, args=(queue,))
 process.start()
 process.join()

def main(start_method):
 multiprocessing.set_start_method(start_method)
 queue = multiprocessing.Queue()

 process = multiprocessing.Process(target=process_function, args=(queue,))
 process.start()
 process.join()

 results = []
 while not queue.empty():
 results.append(queue.get())

 print(f"Results: {results}")
 assert results == [1, 2]

if __name__ == '__main__':
 if len(sys.argv) != 2:
 raise SystemExit(f"Usage: {sys.argv[0]} fork|forkserver|spawn")

 main(sys.argv[1])

I believe that the source of this regression is URL which adds the attribute `is_fork_ctx` to `multiprocessing.Lock()` but doesn't update the pickle methods (`__getstate__()` and `__setstate__()`) so after being serialised and deserialised, the `Lock()` object looses that attribute.

The following patch, adding `is_fork_ctx` to the pickle methods, makes the above work again.

diff
diff --git a/Lib/multiprocessing/synchronize.py b/Lib/multiprocessing/synchronize.py
index 2328d33212..9c5c2aada6 100644
--- a/Lib/multiprocessing/synchronize.py
+++ b/Lib/multiprocessing/synchronize.py
@@ -109,10 +109,11 @@ def __getstate__(self):
 'not supported. Please use the same context to create '
 'multiprocessing objects and Process.')
 h = sl.handle
- return (h, sl.kind, sl.maxvalue, sl.name)
+ return (h, sl.kind, sl.maxvalue, sl.name, self.is_fork_ctx)

 def __setstate__(self, state):
- self._semlock = _multiprocessing.SemLock._rebuild(*state)
+ self._semlock = _multiprocessing.SemLock._rebuild(*state[:4])
+ self.is_fork_ctx = state[4]
 util.debug('recreated blocker with handle %r' % state[0])
 self._make_methods()

[tasklist]
### Tasks

### Linked PRs
* gh-108568
* gh-108691
* gh-108692

Just to add confirmation to what has already been documented here and to document for others using the gcloud cli and finding this in search, I'm on python3.11.5 as well (`Python 3.11.5 (main, Aug 24 2023, 15:09:45) [Clang 14.0.3 (clang-1403.0.22.14.1)]`) and am encountering this error with the `gcloud` cli tools.

A search for the following error message brought me here:

sh
$ gcloud storage cp gs://bucket/file ./ --verbosity debug
DEBUG: Running [gcloud.storage.cp] with arguments: [--verbosity: "debug", DESTINATION: "./", SOURCE:1: "['gs://bucket/file']"]
DEBUG: Starting new HTTPS connection (1): storage.googleapis.com:443
DEBUG: URL "GET /storage/v1/b/bucket/o/data%2Ffile?alt=json&projection=noAcl HTTP/1.1" 200 896
Copying gs://bucket/file to file://./file
INFO: Using high CPU count, single-file workload config.
⠛ERROR: gcloud crashed (AttributeError): 'Lock' object has no attribute 'is_fork_ctx'

This happens regardless of which gcloud version I upgrade or downgrade to:

$ gcloud version
Google Cloud SDK 444.0.0
beta 2023.08.22
bq 2.0.97
core 2023.08.22
gcloud-crc32c 1.0.0
gsutil 5.25

And I can successfully execute the command if I tell gcloud to use an older python version before 3.11.5, e.g.:

sh
$ CLOUDSDK_PYTHON=python3.10 gcloud storage cp gs://bucket/file ./
# (executes just fine)

The fix appears to be merged and backported. Shouldn't this be closed?

It doesn't seem like a fix has been released for Mac yet. I'm still hitting this issue after running `brew upgrade` since it's still using python 3.11.5. I did confirm the workaround provided by @carbocation works correctly.

Correct, the fix version will be 3.11.6 which doesn't exist yet.

looks like python3.11.6 was released today. I haven't tested the fix since I'm waiting for `brew` to support it (to keep my installations manageable).

I can confirm that the problem no longer occurs with python 3.11.6 (tested with `Google Cloud SDK 446.0.1` on `MacOS 13.4`). I had the issue with python 3.11.5


Question: --path is not working?

This issue has been automatically closed because it received no activity for a while. If you think it was closed by accident, please leave a comment. Thank you.

--path is not working
Looks like add command is not working with -p and --path parameters. It's ignoring the given path.

Also it would be good to have a way to set in components.json the path of the generated components.

The `-p` parameter doesn't work for ui components as I describe in this issue #686. In future releases it will be possibile to change the installation folder.

question(cli): override component installation path
@shadcn The `add` command accept the `--path` argument to override the component installation path but it doesn't work for items with the `components:ui` type. In the previous version the cli asked for the installation path, but now is forced to the `ui` subfolder.

 URL

Is this required or not?

I think this is a duplicate of #1156. shadcn has started looking into it.

Right now, the shadcn has a specific folder to install their components, so u gotta adapt your folder structure to receive shadcn components without any problem, in the shadcn installation u can remake your folder structure just pressing "y" on a step.
But to solve your doubt you should visit URL

It looks like there's a new target field that might help here.


Question: [FEAT] Make the advanced options more accessible.

Well, on Mac the application executable is not in the path.
And yes, you do can actually `cd` into .app containers, but I doubt many Mac users know that.
Would require entering something like: `/Applications/Raspberry\ Pi\ Imager.app/Contents/MacOS/rpi-imager --option`

Keyboard combo was also changed "shift" + "ctrl" + "X", to be able to have the same combination on MacOS.
Even though ctrl+alt+x may be easier to remember for some, MacOS lacked "alt".

[FEAT] Make the advanced options more accessible.
I recently heard about the advanced features in passing, and when I went to go image an SD card, I decided that these advanced options would save a lot of time. I then downloaded the imager tool and spent a solid minute clicking the options to find this menu.

After Googling I found this page which ten told me I had to press Ctrl+Shift+X, which isn't obvious.

I feel that this is bad UX, as most people won't be aware of this keyboard shortcuts, even if it would be helpful to them!

I ask for a button somewhere on the main UI (such as another button in the button row or somewhere else)

Yes, second this!

I just found out via a random tutorial... no way of knowing this feature even exists.
And when using it the second time after a while I forgot the "magical" key combination.. so I needed to spend some time again searching to find it.

I would be very nice to have an "advanced" button. Maybe a bit hidden or bind a "are you sure?" modal if you must.

Hey,
If you want this perhaps you can tell me what you think about an unofficial fork, it seems like its what Rpi foundation suggests there:
 URL

The reason I raise this issue is because it is inaccessible to those that
don't previously know of these settings. Making a fork does NOT improve
visibility.

While I'm sure using a terminal to set up WiFi is the "correct" way, for
most users clicking a box in an application on a computer that already
knows the password is easier, especially when they might not have a monitor
& keyboard to attach to the Pi.

On Wed, 19 Jan 2022 at 7:30 pm, Guy Sheffer ***@***.***>
wrote:

You have a point regarding visibility. Unless the fork becomes more common knowledge.
If there are more community OSes onboard that use such a fork, it might make it more widely known. Since it seems keeping the UI simple makes sense if you want to be educational. Rpi Foundation people wrote in the issue there:
#180 (comment) they don't want to do is.
It means there might be enough reason to make a community fork with more advanced options in general.

For example, you could add custom fields as a feature in the advanced tab (domain names, static IPs, you name it, more images to pick from).

There is code in git that will display an extra button (cog icon) if the user selected an image that advertises support for advanced settings.

To advertise support add to the images in your json the option:

"init_format": "systemd"

(or "init_format": "cloudinit" if your image is based on Ubuntu Server)

@maxnet Cool, I understood that is in the works (from our talk here URL .

Just no distributions in the repository that advertise support right now.
But since you are maintaining your own .json file, you do can add the option to make it work for your distro...

I don't have an Ubuntu-based distro on the official JSON atm.
Also my Ubuntu-based distros can also handle wifi settings from the raspbian distro format thanks to CustomPiOS network module. So if you pick them as a custom images advanced settings for wifi do work.

If you have RPI OS based distros the setting would be "init_format": "systemd"

(instead of cloudinit)

OS customisation is now accessible without a keyboard shortcut (and Ubuntu server is in the standard OS list with customisation enabled)

Make advanced options discoverable
I submitted two issues (#178 and #179) to do automatic setup of SSH and wifi. Turns out that feature already exists, which is fantastic!

Unfortunately it's completely undiscoverable. There's no way to know it exists unless you magically already know.

There should be some visual indication that the option is available - "do you want to set up advanced options?" sort of thing.

I have to say you guys make a lot of disappointingly user-hostile decisions. You don't let users resize the window (a very standard feature), you hide the option to set the freaking wifi password. That is so beginner unfriendly!

If by "discussed" you mean "almost everyone agreed it was bone-headed", then sure. Look at the comments from your users!

I'd also like to reply to one of the two (that I counted) replies in favour:

Well for a start you literally just forgot the shortcut!, but secondly how are you supposed to remember the shortcut if you never even knew it existed?

This goes against basically every UX principle ever. I could understand it if the options were super dangerous, but we're talking about setting up wifi here! What world are you living in where you think people with a Raspberry Pi are intimidated by wifi?

Anyway I'll leave you to your crazy designs.

@maxnet / @ghollingworth Is it worth mentioning the magic advanced-mode key-combo in URL ?

If you actually want users to know about this very useful feature then just put a button for it in the app. Very few people will read the Readme but everyone uses the app.

If you don't want users to know about this feature then you should think hard about why, because it's clearly insane to write a feature that you don't want anyone to actually discover.

Yes, everybody working at Raspberry Pi is clearly insane :stuck_out_tongue_winking_eye: Welcome to the madhouse :upside_down_face:

It's not even been 2 weeks and I just had to look up the key combo already. To do that I had to first find the blog post. I'll accept I have a poor memory but that aside, this was by no means resolved in the blog comments. I fully appreciate these extra options might be overwhelming/confusing to our newest users and we absolutely don't want that, but I think there is a middle ground here. Could we discuss what that might look like?

The shortcut sort of was the middle ground.
Between team #we-are-targetting-beginners rejecting the concept of a button, and team #experts coming up with a command line option (which is non-trivial to use on say Mac OS X).

OK, thanks for that explanation regarding Mac.

So to clarify, is a button with a very clear warning really off the table? Such a button could be small and placed in the bottom corner to reduce the chance of being accidentally pressed. Something like this:

Screenshot from 2021-03-31 14-14-04

If someone still feels this could lead to anyone accidentally changing the advanced options, what if one of the advanced options was to always show this button? So then you only have to remember/lookup the magic shortcut once. If that is still somehow disagreeable, you could have the button hidden unless the `--show-advanced-options` command line option is used (for the benefit of non-Mac OS users only).

I'm really struggling to see how this adjustment makes the tool noticeably more confusing for new users. And it's much better for those other users who do want to easily enable SSH, change the hostname, configure WiFi etc.

So, would you accept a pull request with such a change? If not, what could we adjust to make it more apealing to those in charge here?

For the moment we've already made our decision about how we're going to support this functionality. As said many times before we do not want to change the user interface at the moment since this is the easiest way for people to image SD cards.

When an easier way appears we may then make this a more complex interface for more advanced users, but until then we will not be changing it.

You are free to build and distribute your own version (as long as you remove the Raspberry Pi logos and wordmark from the front page.)

Disappointing but of course that's your decision to make. Thanks for taking the time to reply.

Hey,

@ghollingworth

Er - I am actually considering making my own version, since there are about 6 standing ``[OS Request]`` issue with no answers (filter to requests).
Does it make sense to you guys there? I don't want to step on anyone toes, though it seems like all other users in the thread might welcome a "unofficialpi" fork.
I am also in the works of adding json generation in CustomPiOS (related URL ). And knowing what is the stand on that would help me decide if a fork is worth having.

(I was actually got here thinking we could join efforts and make a smoother CI/CD using github actions or something more accessible)

In general, those requests have sat idle while I think about how best to put them in. In the end, the more images that do the same thing (home automation or 3D printer for example), I'm just going to add in a secondary list. For example:

Other Specific Purpose OSs
 -> OctoPi
 -> Other 3D printer OSs
 --> ABC
 --> Simply etc

Hey, so I started looking in to making a fork, for now It already uses its own json list and merges ones in another repo:

 URL

And it takes it from:
 URL
Which already has a github action to merge the official .json and add on top of it.

If I get Ci/CD for this running we could have a less restricted and more advanced version of rpi-imager. Perhaps even make it support other SBCs.
It could also be a space to see what works and then merge it up to mainstream.

Nice work guysoft! @Raspberry Pi why did you delete his message? If there's a good reason then you should probably say before people make assumptions. If there isn't then I would strongly consider reinstating it ASAP before people realise!

Hey, just to update that I spoke to Gordon and all is good. No need to make any assumptions :)
I might start working on a "packed with distros rpi-imager" once I get OctoPi and FullPageOS RCs out.


Question: Typography: Theme defined font sizes are no longer working?

It seems we're keeping this on the WordPress 6.5 Editor Task board intentionally in the No Status column in hopes that it'll be addressed for 6.5. It does seem like a critical issue for many builders. I see it as In Progress on the Automattic team Ignite's project board. @ajlende do you have any updates on this, please?

Typography: Theme defined font sizes are no longer working
### Description

It looks like the font sizes a theme defines in its theme.json are no longer working.

| without plugin | with plugin |
| ----------- | ----------- |
| localhost local_ (12) | localhost local_ (13) |

For example, in the theme, `--wp--preset--font-size--medium` should be `1.05rem` but with Gutenberg plugin, the size is `clamp(14px, 0.875rem + ((1vw - 3.2px) * 0.625), 20px)`

I can see this issue with other themes too.

### Step-by-step reproduction instructions

1. Open a homepage in the browser.
2. Deactivate Gutenberg trunk
3. Open the homepage in a new tab
4. Notice the difeerence

### Screenshots, screen recording, code snippet

_No response_

### Environment info

- WP 6.4.2
- Gutenberg Trunk

### Please confirm that you have searched existing issues in the repo.

Yes

### Please confirm that you have tested with all plugins deactivated except Gutenberg.

Yes

Pinging @richtabor and @annezazu b/c IMO this is a pretty big deal (and breaking change).

To further illustrate what is happening, see below.

**What font sizes should be in my Powder theme:**

image

**What is output with Gutenberg 17.5 active:**

image

Moreover, Gutenberg trunk seems to exacerbate the problem with even more unexpected results and—in this case—breaking changes on the front end.

Perhaps this is related (or has similar behavior) to URL

Also pinging @carolinan as she was involved in the issue I mentioned.

@ajlende, could this be a side-effect or regression from work introduced in #56661 ? Can you take a peek, please?

@bgardner @colorful-tones, thanks for adding more context.

 It'd be great if this issue gets fixed quickly. It blocks theme creations if you use the trunk :/

I'll escalate where I can too. Thank you for the ping and raising the alarm.

Can’t wait to see what you come up with @ajlende, as Gutenberg 17.6 RC1 now has breaking changes on the front end with font sizes.

Any progress on this? The first image comparison with TT4 might have not illustrated the issue very much because the difference was subtle. The extent of the disturbance depends on the design of the theme, and this example might illustrate the issue better The left is how it should be and the right is the current status.

Screenshot 2024-01-29 at 13 47 04

I am far from an expert in how those properties are handled in Gutenberg, but maybe changing the `prevent_override` to `false` solves the problem: link.

prevent_override => false

I've tested that and, making that change, it seems to work for my case:
* If I have a theme.json with sizes with the same slug as the default ones ("small", "medium", "large"...) the value from the theme takes precedence over the default value in Gutenberg.
* If I remove the `fontSizes` from the theme, they still appear now with the values provided by Gutenberg.

Is that what we want? If that should be the solution I can work on a pull request.

@SantosGuillamot Thanks for taking a look! I've also been working on this and commented in the PR that caused the issue (#56661). It'll be helpful if you can review when I get it working.

Great to hear you are working on that 🙂

Oh, that's a pity. I can test whatever the solution is, although I'm not gonna be available until tomorrow.

Noting that while the front end is better/fixed, the font sizes in the editor (Typography > Size) are still being hijacked by core sizes. In other words, the screenshots from my comment above are still in an issue.

cc: @annezazu for visibility and @iamtakashi to confirm if seeing the same thing.

To re-iterate, these were just taken with/without Gutenberg 17.6.0-rc.3 active.

**Gutenberg not active:**

image

**Gutenberg active:**

image

Just reopening if this is still an ongoing issue and not fully corrected with the latest change.

@bgardner - Just to help me understand the current issue:

From what I'm seeing in your latest screenshots is that the core sizes are listed above yours. If that is the only problem, there's a separate issue for that: URL

Or is it that your custom sizes are not working still?

@justintadlock I see that this exists: URL

The problem is that in the editor, where there are font sizes names common with core/default themes and my Frost/Powder themes (Small, Medium, Large, and Extra Large in this case) the drop down menu shows the core values and not my theme values. With the revert in URL the issue is no longer on the front end out of the box—just when text is given any of those sizes listed above (in which case it becomes a front end problem.)

In English: if a theme declares font sizes that are identical to the ones in core, core should defer altogether (front end, back end, site editor) to the theme.

For me I have turned off every style, but with Gutenberg 6.4.3 (since 6.4.0 iirc) font size is back

`theme.json`:

json
{
 "$schema": " URL
 "version": 2,
 "settings": {
 "appearanceTools": false,
 "typography": {
 "defaultFontSizes": false,
 "customFontSize": false,
 "lineHeight": false,
 "dropCap": false,
 "fluid": false,
 "fontStyle": false,
 "fontWeight": false,
 "letterSpacing": false,
 "textDecoration": false,
 "textTransform": false,
 "fontFamilies": [],
 "fontSizes": []
 }
 },

The font size control is back, making the whole "Styles" tab appear:

image

Since this issue has been reopened, I will also change the project board status from "Done" to "ToDo".

I think the original issue caused by URL which was resolved when we reverted in #58456.

The original issue ( URL is still open, as is this other one about duplicate font sizes: URL

I think these two issues cover all the currently known bugs, so I'll close this one in favour of them.

Global styles: Default font sizes are added
### Description

I created a blank theme and added custom font sizes.
This works intended in the editor, but in the Global styles, default font sizes are added in the drop down (S, M, L, XL)

### Step-by-step reproduction instructions

1. Copy the nearly blank theme.js from below
2. Go into Global Style → Typography → Text
3. See in the drop down the additional "Small, Medium, Large, Extra Large" font sizes, that shouldn't be there.

 theme.js

javascript
{
 "settings": {
 "appearanceTools": true,
 "color": {
 "background": true,
 "custom": true,
 "customDuotone": true,
 "customGradient": true,
 "defaultDuotone": false,
 "defaultGradients": false,
 "defaultPalette": false,
 "duotone": [],
 "gradients": [],
 "palette": [],
 "text": true
 },
 "layout": {
 "contentSize": "620px",
 "wideSize": "1000px"
 },
 "shadow": {
 "defaultPresets": true,
 "presets": []
 },
 "spacing": {
 "customSpacingSize": true,
 "spacingScale": {
 "increment": 1.5,
 "mediumStep": 1.5,
 "operator": "*",
 "steps": 7,
 "unit": "rem"
 },
 "spacingSizes": [],
 "units": ["%", "px", "em", "rem", "vh", "vw"]
 },
 "typography": {
 "customFontSize": true,
 "dropCap": true,
 "fluid": true,
 "fontFamilies": [
 {
 "fontFamily": "-apple-system, BlinkMacSystemFont, 'Segoe UI', Roboto, Oxygen-Sans, Ubuntu, Cantarell, 'Helvetica Neue', sans-serif",
 "name": "System Font",
 "slug": "system-font"
 },
 {
 "fontFace": [
 {
 "fontFamily": "Barlow",
 "fontStyle": "italic",
 "fontWeight": "700",
 "src": ["file:./assets/fonts/barlow_italic_700.ttf"]
 },
 {
 "fontFamily": "Barlow",
 "fontStyle": "normal",
 "fontWeight": "700",
 "src": ["file:./assets/fonts/barlow_normal_700.ttf"]
 },
 {
 "fontFamily": "Barlow",
 "fontStyle": "italic",
 "fontWeight": "600",
 "src": ["file:./assets/fonts/barlow_italic_600.ttf"]
 },
 {
 "fontFamily": "Barlow",
 "fontStyle": "normal",
 "fontWeight": "600",
 "src": ["file:./assets/fonts/barlow_normal_600.ttf"]
 },
 {
 "fontFamily": "Barlow",
 "fontStyle": "italic",
 "fontWeight": "400",
 "src": ["file:./assets/fonts/barlow_italic_400.ttf"]
 },
 {
 "fontFamily": "Barlow",
 "fontStyle": "normal",
 "fontWeight": "400",
 "src": ["file:./assets/fonts/barlow_normal_400.ttf"]
 }
 ],
 "fontFamily": "Barlow",
 "slug": "barlow"
 }
 ],
 "fontSizes": [
 {
 "name": "Display xl",
 "slug": "Display-xl",
 "size": "96px",
 "fluid": {
 "max": "96px",
 "min": "52px"
 }
 },
 {
 "name": "Display lg",
 "slug": "Display-lg",
 "size": "72px",
 "fluid": {
 "max": "72px",
 "min": "46px"
 }
 },
 {
 "name": "Display md",
 "slug": "Display-md",
 "size": "60px",
 "fluid": {
 "max": "60px",
 "min": "42px"
 }
 },
 {
 "name": "Display sm",
 "slug": "Display-sm",
 "size": "52px",
 "fluid": {
 "max": "52px",
 "min": "40px"
 }
 },
 {
 "name": "Heading 1",
 "slug": "Heading-1",
 "size": "48px",
 "fluid": {
 "max": "48px",
 "min": "36px"
 }
 },
 {
 "name": "Heading 2",
 "slug": "Heading-2",
 "size": "36px",
 "fluid": {
 "max": "36px",
 "min": "30px"
 }
 },
 {
 "name": "Heading 3",
 "slug": "Heading-3",
 "size": "28px",
 "fluid": {
 "max": "28px",
 "min": "26px"
 }
 },
 {
 "name": "Heading 4",
 "slug": "Heading-4",
 "size": "24px",
 "fluid": {
 "max": "24px",
 "min": "22px"
 }
 },
 {
 "name": "Heading 5",
 "slug": "Heading-5",
 "size": "20px",
 "fluid": {
 "max": "20px",
 "min": "19px"
 }
 },
 {
 "name": "Heading 6",
 "slug": "Heading-6",
 "size": "18px",
 "fluid": {
 "max": "18px",
 "min": "17px"
 }
 },
 {
 "name": "Paragraph lg",
 "slug": "paragraph-lg",
 "size": "18px",
 "fluid": {
 "max": "18px",
 "min": "17px"
 }
 },
 {
 "name": "Paragraph base",
 "slug": "paragraph-base",
 "size": "16px",
 "fluid": {
 "max": "16px",
 "min": "16px"
 }
 },
 {
 "name": "Paragraph sm",
 "slug": "paragraph-sm",
 "size": "14px",
 "fluid": {
 "max": "14px",
 "min": "14px"
 }
 },
 {
 "name": "Info md",
 "slug": "Info-md",
 "size": "12px",
 "fluid": {
 "max": "12px",
 "min": "12px"
 }
 },
 {
 "name": "Info sm",
 "slug": "Info-sm",
 "size": "10px",
 "fluid": {
 "max": "10px",
 "min": "10px"
 }
 }
 ],
 "fontStyle": true,
 "fontWeight": true,
 "letterSpacing": true,
 "textDecoration": true,
 "textTransform": true
 },
 "useRootPaddingAwareAlignments": true
 },
 "styles": {
 "spacing": {
 "padding": {
 "bottom": "0px",
 "left": "16px",
 "right": "16px",
 "top": "0px"
 }
 }
 },
 "templateParts": [
 {
 "area": "header",
 "name": "header"
 },
 {
 "area": "footer",
 "name": "footer"
 }
 ],
 "version": 2,
 "$schema": " URL
}

### Screenshots, screen recording, code snippet

| Block styles (as it should be) | Global styles (with additonal font sizes) |
|--------|--------|
| <img width="160" alt="image" src=" URL | <img width="160" alt="image" src=" URL |

### Environment info

- WP 6.2.2
- GB 16.1.0

### Please confirm that you have searched existing issues in the repo.

Yes

### Please confirm that you have tested with all plugins deactivated except Gutenberg.

Yes

Hi
I copied the code example from the issue and used it to replace the theme.json in Twenty Twenty-Three.

- 6.2.2 only: No issue
- 6.2.2 and 16.1.0: Additional font sizes as described
- 6.3-beta2-56121: Additional font sizes as described

@annezazu Can you please add this issue to the Polish board ? Cheers!

Just bumping. I indeed noticed this as well. I'm seeing the default sizes appearing in the editor alongside custom ones as well as in the DOM as variables. I can't actually find a way to remove them. I don't think this happened in the past so I'm not sure what changed. I'm running 6.4.1 and the Gutenberg 17.0.2

Also noting that this can cause a known issue where two type sizes can conflict with eachother. If two type sizes have the same value then selecting one will cause the block to revert to the first matching style that appears in the list. Normally this isn't a big issue, unless the default styles don't disappear when you define custom ones. The default styles appear first in the list so selecting a custom style with the same value forces it to select the default.

I'm seeing the same settings in both places (WP 6.4 + GB trunk); however, the four default font sizes ('Small', 'Medium', 'Large', 'Extra Large') appear first in the block styles, but last in the global styles. In other places where Gutenberg provides default values for something we add a `default*` option to disable showing those settings, but I don't see one for font sizes right now.

So, it sounds to me like there are two parts to solving this fully.

1. Add a `defaultFontSizes` option that allows for the default size options to be hidden.
2. Ensure that the order of the default, theme, and user sizes are the same in both places when they are all enabled.

This should work as a quick fix for disabling the default font sizes, but I'm pretty sure the CSS for them won't be printed either:

php
add_filter('wp_theme_json_data_default', function($theme_json) {
 $data = [
 'version' => 2,
 'settings' => [
 'typography' => [
 'fontSizes' => []
 ]
 ]
 ];

 return $theme_json->update_with($data);
});

@justintadlock Can confirm the filter above is working perfectly and as expected. CSS is printed and Small, Medium, Large, and Extra Large sizes show correct values in the Typography > Font Size menu in post editor as well as correctly on the front end when chosen.

@carolinan Agreed. I was merely confirming it worked to see if it helped inform a permanent fix in core.

I have removed this from the Polish board is it's too big to go into 6.5.

@scruffian I'm a bit confused by this last comment. Can you clarify? I've added it to the 6.5 board to be safe as it's clearly impacting theme authors and seems important to get fixed. Ideally, this is resolved for 6.5 but I definitely want to understand more if/why that might not be feasible.

We put in a fix for this, but it ended up creating worse bugs, so we took it out again. We are still working on a fix, but it's going to take longer than we have before the cut off. I hope it will make it into a Gutenberg release soon :)

I agree with @carolinan on this, it is inconvenient to ask to touch two places to configure the theme but at the same time I think it is nice to give the opportunity to confiugure the theme dynamically.

I've seen some options line `blockGap` accepting three values, `false`, `true`, `null` triggering different results. Could be the same for the collections, something like `false` deactivate the feature, `[]|{}` empty object add nothing more. Indeed we cannot pass an empty collection to signaling the theme we want any option because it would make difficult to address the cases where someone might want to edit the value via a filter giving dynamic data which might be empty or not while at the same time they want to keep the current theme / core configuration.

We can't expect theme developers to add this filter as a long term solution.

I agree with @carolinan on this, it is inconvenient to ask to touch two places to configure the theme but at the same time I think it is nice to give the opportunity to confiugure the theme dynamically.

I've seen some options line blockGap accepting three values, false, true, null triggering different results. Could be the same for the collections, something like false deactivate the feature, []|{} empty object add nothing more. Indeed we cannot pass an empty collection to signaling the theme we want any option because it would make difficult to address the cases where someone might want to edit the value via a filter giving dynamic data which might be empty or not while at the same time they want to keep the current theme / core configuration.

### Update
Sorry, I've not took into account the theme override the core configuration, disabling via the same option wouldn't work because there'll be no way to add more options. (edited)

@annezazu To clarify some more, we want to fix this issue by adding a `defaultFontSizes` option in theme.json which matches how many other presets work. And we _thought_ we had it working in #56661.

Unfortunately, the behavior of having that setting either enabled or disabled is different from the current implementation (merging default options with theme options). This is the "worse bug" that @scruffian mentioned—old themes looked different after the change. To fix that issue, we need to bump the theme.json version so old (theme.json v2) themes can continue to work as they currently do and new (theme.json v3) themes can start using the new defaults.

Increasing a theme.json version isn't a small change. And this is the first time that we're changing a default value, so there's no precedent set for how to do it. We want to get it right because there's two other options that could be updated the same way.

You can see my progress in #58409. The change already seemed big enough that we didn't want to rush it for 6.5.

This isn't going to make it to 6.5 as stated above :( I left it for a bit to see if we could make progress since, as you noted, it's an important issue but it's not going to happen 24 hours before beta 1. Punting now.

@justintadlock So you're saying that the font sizes will be broken in the site editor for any theme that uses slugs similar to core?

Yes.

This has been an issue for several major releases and no one has a fix for it yet that doesn't break other things. Given the complexity of the issue, it's unlikely to be fixed in time for 6.5. Of course, if any contributor has time for a PR to fix in the next 24 hours, that could change. I'm sure plenty of folks would be willing to test.

@justintadlock This issue has not existed (at least what I am experiencing URL until recently. The current version 6.4.3 is not experiencing this issue, and if ignored, this will become a major disaster for users of these themes. I am not sure at this point if anyone understands the severity of it, and I will glad Zoom you (or @annezazu) or whomever to show exactly what is happening.

@bgardner - This specific ticket was created for Wordpress 6.2.2 and Gutenberg 16.1.0, so I want to make sure it's clear we're talking about punting something that's been understood to be an ongoing thing.

Not sure if #57889 should be reopened at the moment, especially if it's a new issue introduced since WP 6.4. In that case, it should definitely be addressed for 6.5. Anyway, pinging some folks to see what we can get done. :)

@justintadlock I understand. The original issue that was created ( URL was created at 6.4.2, and that is where I chimed in. It has since been closed, but the problem still exists. I think it needs to be re-opened (again) and made a priority for 6.5.

Because this issue touches all of my themes, Ollie by Mike McAlister, even @ndiego's personal theme. Not to mention the (possible) 100's of themes that are using theme sizes/slugs similar to core.

Circling back to ack that I Zoomed with @annezazu and @ajlende to walk through what I reported in URL More to come.

I tried to describe the problem we still see in this thread. Sorry, if it wasn't clear.

@scruffian I'm not sure the fix was reverted, we reverted a change that create this issue.

I think it's okay to close this now.

I figured it was still in a broken state because URL had fixed it, but was reverted in URL causing it to be an issue again. URL was intended to reimplement URL and solve the issue again. However, URL which was the origin of this bug was reverted in URL fixing this issue. 😵💫

I still have the PR for adding the `defaultFontSizes` option in URL which I would still appreciate some reviews on 🙂 But I will be closing this issue now.


Question: profile param does not working with AWS SSO config?

lets move any discussion here to the aws repo and close this one. Check out the reply here: URL for a working workflow with secrets

profile param does not working with AWS SSO config
I am able to use `load_aws_credentials()` with my default profile but it does not work when I specify a different profile, e.g. `load_aws_credentials('prod')`

@bobcolner thanks for reporting. SSO configs may not yet be fully supported

Hey @samansmink!
I've played around and using sso while specifying the profile still doesn't work.
BUT when doing this :

CREATE SECRET (
 TYPE S3,
 PROVIDER credential_chain,
 CHAIN 'sso',
 PROFILE 'my-profile'
 )

 This does work. So, I guess we missed a small thing in the extension 🤔?
 Maybe worth updating the documentation to mention how to use `sso` , I've heard a couple of users requesting this

This all seems somewhat broken on v0.10 on Linux:
* `CALL load_aws_crendentials();` no longer works on `v0.10.0 20b1486d11` for me when `AWS_PROFILE` is set. This is regression from `v0.9.2 3c695d7ba9`. (Although even 0.9.2 is not reliable - see #10.)
* `CREATE SECRET` does not work for me when specifying the profile name - it returns `Success=true` from the call, but I then get a 403 error when the S3 `GET` request is made
* If I set the `AWS_PROFILE` env var and use `CREATE SECRET`, omitting the profile parameter, it fails in the same way (and also if you omit the `CHAIN` parameter too)

I can't make either `CALL load_aws_credentials()` or `CREATE SECRET` work on v0.10. If I run `aws configure export-credentials --profile foo` and manually set the legacy `s3_access_key_id`, `s3_secret_access_key` and `s3_session_token` config variables then everything works properly, so there's nothing wrong with the current SSO session, just the wiring here in DuckDB.

Hey everyone, thanks for reporting. I will try to set up some SSO config in our CI testing environment to properly test the SSO login method and fix any issues.

Just for clarity here, @herebebeasties Did you also try the create secret syntax as described by @mehd-io?

Afaict the workflow described by @mehd-io works. I will PR this to the docs.

Since `load_aws_credentials` is now deprecated, I did not test that and will not fix that if its broken

I will leave this open until this is properly tested in the aws extension ci

Yes. It did not work for me - see the bullet points in my above message. What I've put in my above message is accurate.

@herebebeasties

Could you provide a reproducible example for the `CREATE SECRET` flow, describing the exact steps you've taken and the contents of your ~/.aws/config file? otherwise its a bit hard for me to reproduce this

I'm not really sure what a "reproducible example" would look like here beyond what I've already described, other than scripting up the creation of a whole AWS environment, S3 bucket, AWS policy, role, policy mapping, Microsoft Azure AD domain creation, SSO setup, etc. which would be pretty complex and time-consuming. Do you have something smaller and more specific in mind?

I'll try to find some time to have a stab at debugging why this is failing in gdb, as tbh it's probably easier to do that than the above.

Well, for example, I ran:

`aws configure sso`

now my `~/.aws/config` looks like

[default]
region=eu-west-1

[profile duckdb-sso-test]
sso_session = ...
sso_account_id = ...
sso_role_name = ...
region = eu-west-1

[sso-session duckdb-sso-test]
sso_start_url = ...
sso_region =...
sso_registration_scopes = ...

Then to login I run:

aws sso login --profile=duckdb-sso-test

Now I get a pop up in my browser to login, which I do.

Then in DuckDB i run:

SQL
CREATE SECRET (
 TYPE s3,
 PROVIDER credential_chain,
 CHAIN 'sso',
 PROFILE 'duckdb-sso-test'
);

And now my auth works fine, and when I query `SELECT secret_string FROM duckdb_secrets()` I get:

name=sso;type=s3;provider=credential_chain;serializable=true;scope=s3://,s3n://,s3a://;endpoint=s3.amazonaws.com;key_id=redacted;region=eu-west-1;secret=redacted;session_token=redacted

Alternatively, If you would give a stab at debugging this, that would be super useful for sure.

That's exactly the same skeleton as my set-up and yet it doesn't work for me. I'll have a look in a debugger.

Thanks! Be aware of the fact that I had to rerun the `aws sso login --profile=duckdb-sso-test` after some time. Also, DuckDB stores the raw temporary credentials in the secret, so when the session token expires, the secret is not automatically updated and needs to be manually re-created

interesting, I have the same problem with duckdb under linux

what worked in the end:

1. write tmp credential to env: `$(AWS_PROFILE=dev-sso aws configure export-credentials --format env)`
2. start duckdb
3. load credential: `CALL load_aws_credentials();`
4. set the region: `SET s3_region = 'eu-central-1';`
5. select data: `SELECT * FROM 's3://bucket/path/file.parquet';`

Using the CONFIG storage above, it returns "success" but does not store any secret:

D CREATE SECRET (
 TYPE S3,
 PROVIDER CREDENTIAL_CHAIN,
 REGION 'eu-central-1',
 PROFILE 'dev-sso',
 ENDPOINT 's3.eu-central-1.amazonaws.com'
 );
100% ▕████████████████████████████████████████████████████████████▏
┌─────────┐
│ Success │
│ boolean │
├─────────┤
│ true │
└─────────┘

D SELECT secret_string FROM duckdb_secrets();
┌────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────┐
│ secret_string │
│ varchar │
├────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────┤
│ name=__default_s3;type=s3;provider=credential_chain;serializable=true;scope=s3://,s3n://,s3a://;endpoint=s3.eu-central-1.amazonaws.com;region=eu-central-1 │
└────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────┘

I put up a PR (#49) to support SSO auth using the new method. It works on my machine :tm:

I will close this issue, it works now on at least 2 machines ;) Feel free to open a new issue if sso problems persist

support loading credentials set by `aws sso login`
`aws sso login` is a common path for users to log into their AWS console. The extension does not currently extract tokens from `~/.aws/sso/cache` where authentication token is cached to disk through this path. User needs to manually run `aws configure` to set up their `~/.aws/credentials` instead for the extension work. It would be great if support for the common path of `aws sson login` can be supported also.

This setup seems to work for me:

[default]
sso_session = my-sso
sso_account_id = 123456789123
sso_role_name = SomeRole
region = us-east-1
output = json

[sso-session my-sso]
sso_region = us-east-1
sso_start_url = URL
sso_registration_scopes = sso:account:access

Not sure if something was changed since this issue was opened.

Doesn't work here but I have a different setup with:
`[default]
sso_session = my-sso
sso_account_id = 123456789123
sso_role_name = SomeRole

output = json

[sso-session my-sso]
sso_region = us-east-1
sso_start_url = URL
sso_registration_scopes = sso:account:access`

Also not working for me with same config as @rairaman
I'm on `v0.9.2 3c695d7ba9`, linux ubuntu 22.04. I tried both zsh and bash, and even python with `duckdb.sql()`.
Nothing works.
A colleague of mine has it work on his mac, his aws profile looks the same :shrug:

After building myself, it works:

./build/release/duckdb
v0.9.3-dev1243 6af3519ec1
D call load_aws_credentials();

Gives me a nice table full of credentials, while

duckdb
v0.9.2 3c695d7ba9
D call load_aws_credentials();

Gives me an empty table.

For me at least, if you have just done an `aws sso login` then it works. If some time has gone by then it doesn't.

"Ah, well your session must have expired!" I hear you say. But no; `aws s3 ls` works just fine. So something else going on here.


Question: Introduce common undo functionality and prevent accidental cell deletion?

This is fixed in main and will be available in next Insiders (tomorrow) and the next Stable.

Introduce common undo functionality and prevent accidental cell deletion
My request concerns the same as this one (closed issue from 2+ years ago): URL

It seems to be a frequently occurring problem until today, and a reason for people not to use VSCode for Jupyter notebooks. See for example: URL

It happens all too often: once a cell is not in focus, and one hits the common key combination to undo the last change (on Mac: Cmd + z), a cell gets deleted. Despite the suggested "solution" in issue 1998 (just hitting the key "z" when not focused on a cell), this does more of the same: it deletes cells!
Once a cell is deleted, undoing via menu or via Cmd + Shift + P (command palette), deletes just more cells.

So basically, with this extension, undoing in any way means deleting. And deleting can't be undone.

I use VSCode for programming a lot. Just removing the common undo key combination is not an option for me.

Also, I think there really ought to be a working solution to undo an accidental deletion, because it seems to be the norm in any application these days to have the ability to undo the last change (deleting a cell is a change as well).

Besides the obvious: it would be nice to undo the last edit, whether a cell is active or not!

Has there been any developments in this regard?

Exactly @roblourens it's about the undo operation (no matter how it's called - "z", "cmd + z", from the command palette, from the menu), and that it doesn't undo the last action but deletes a cell, if the cell is not focused.

BUT: this also happens when there IS focus on a cell, one scrolls up to another part of the notebook, and types. Usually, VSCode jumps back to where the cursor is still active (as do other apps). In jupyter mode, the last active cell is deleted.
And yes, if one "panicks", then indeed, more cells are added to the "kill list" ;)

_I use Local history, but it does not record any history when working on jupyter notebooks, and I found no config to change that. Also jupyter checkpoints don't get created (I saw another issue about that - I believe that is related to my M1 device though, so let's not entangle issues)._

I think there are the following two problems here:

1. the undo action behaves very abnormally compared to any other modern app, when a cell is not active
2. the same applies if there theoretically IS focus on a cell but the cell is scrolled out of the viewport
3. once undo accidentally deletes a cell, there's no way of restoring it

It is undoing the last notebook-level action, it's undoing the cell addition. If you "redo" it puts the cell back. If you delete a cell like with cmd+backspace, then undo, it undoes the delete.

Your problem 2 is annoying, yes, we might be able to do a better job with the focus state of cells that are scrolled out of view (and thus not actually rendered and focused).

So for 1, I believe it is consistent, maybe just not obvious since the cell "add" may have been a long time ago, so you don't realize you're undoing it. And 3, it can be restored by "redo" which makes sense, this is reversing the action you just did.

Also, the local history does seem very inconsistent.

Thank you for the hint about redoing, I wasn't aware! That indeed solves (3).

Let's use URL to track problem 1&2 as both are talking about the same that it's not clear that users are undoing an operation they might not be aware of.

Type in an invisible cell in edit mode, it goes to command mode
### Applies To

- [X] Notebooks (.ipynb files)
- [ ] Interactive Window and\/or Cell Scripts (.py files with \#%% markers)

### What happened?

When trying to type in a cell (in edit mode for sure), but the cell is invisible, it automatically goes into command mode.

Typing to an invisible cell is common:
- user pastes many lines of code at the end of a cell;
- user scrolls to check other part of the notebook.

This can be dangerous as user may hit `d` multiple times while typing, so cells are deleted.

The expected behavior is: when cell is invisible while in edit mode, typing should let notebook jump back to make it visible and it's kept in edit mode.

### VS Code Version

1.72.2

### Jupyter Extension Version

2022.9.1202862440

### Jupyter logs

shell
(no special log for this issue)

### Coding Language and Runtime Version

_No response_

### Language Extension Version (if applicable)

_No response_

### Anaconda Version (if applicable)

_No response_

### Running Jupyter locally or remotely?

Local

As far as I know, this would only affect when the cell itself is scrolled out of the viewport. This case

when the cursor is out of the viewport but the cell itself is still rendered, should be fine.

In fact, it's not fine: cell will be out of view, too.

I created this issue because I just deleted 2 cells by accident as I described.

Also, IMAO I don't agree it's a feature request, this behaviour can confuse user and cause loss of data; hope you can reconsider its priority. :)

I agree it's actually a bug, when the previous focus is in the cell out of viewport, users would expect typing can bring the cell into view, or at least just typing instead of running keybindings in command mode.

I am having problems with this aspect of the Jupyter extension also, I have luckily only deleted cells a couple of times.

With a fix underway, are there any suggested workaround settings that could perhaps at least mitigate the entering of command mode from an newly unrendered cell?


Question: Tests / Docs / Docs fails on old PRs with blurb?

It is not clear to me whether to backport, so I did not, but this could still be done. Closing now.

Tests / Docs / Docs fails on old PRs with blurb
The tests for PR #8150 were all green. When I added a blurb for #8150, Tests / Docs / Docs failed because of the 4-digit number.
 URL

The error occurred during the sphinx html build, but I have no idea where the misinformation is stored or what tracker this properly belongs on.

The full message:

It doesn't work with _PR_ numbers.

It's expecting either a GitHub _issue_ number, the lowest such began at 32426, from when we transferred issues over from URL

Or use the other format to specify a BPO ID.

I think the easiest thing is to create a new issue for the PR, and use that issue number.

There was no bpo issue: in the initial message, Ammar said "(Can be marked with skip issue and skip news as it pertains to the build process)" but I decided that a visible addition possibly used by non-core devs (to run coverage) should be mentioned. I opened #106690.

Add a .coveragerc file to the CPython repository
Check branches both ways and ignore code that should be excluded from coverage. This includes some special directives for IDLE.

This file is used by the coverage package. It can be used where it is, in the repository directory, or copied elsewhere, depending on how one runs coverage.

### Linked PRs
* gh-8150


Question: vcpkg provided FluidSynth sounds incorrect compared to Meson wrap?

This is not a bug. What you hear in `vcpkg.ogg` is called chrous - ever heard of?

The soundfont you use enables chorus for almost every instrument. So, even if the MIDI doesn't use chorus, the soundfont will!

If you don't like it, either use a different soundfont or disable chorus completely in fluidsynth.

The fix implemented by #1339 does ensure that chrous will now be heard in both, single and double precision builds.

There is no "gold standard". IIRC, CMake had double precision mode enabled by default. The old and long gone autotools buildsystem had single precision as default. My advice: Use single precision. It's less CPU expensive, better auto-vectorizable and sounds exactly the same. (Well, unless one encounters fancy numeric bugs like #1331.)
Perhaps I'll change the default to single precision in the future.

vcpkg provided FluidSynth sounds incorrect compared to Meson wrap
### Are you using the latest DOSBox Staging version?

- [x] I have checked and I'm using the latest release.

### Different version than latest?

_No response_

### What operating system are you using?

Linux x86_64

### If other OS or Linux, please describe

Debian 12

### Are you running DOSBox Staging on a native OS?

- [x] I am running Staging on a native OS

### Relevant hardware info

_No response_

### Have you checked that no other similar issue already exists?

- [x] I have searched and have not found similar issues.

### A clear and concise description of what the bug is

It sounds like it's underwater or something. I also get the bad sound with my system (Debian provided) FluidSynth. Only the Meson wrap sounds correct.

Bad - System 2.3.1
Good - Meson wrap 2.3.3
Bad - vcpkg 2.4.2

fluidsynth-recordings.zip

Soundfont I'm using:

SC-55.SoundFont.v1.2b.sf2.zip

### Steps to reproduce the behaviour

Explain in detail how to reproduce the bug. Assume no familiarity with the software.
1. Use Sound font linked above with FluidSynth
2. Play first level of Doom using General MIDI
3. Test with meson using `meson setup --force-fallback-for=fluidsynth` - Sound correct
4. Test with CMake using vcpkg - Unlock hidden Doom underwater level

### Download URL of affected game or software

_No response_

### Your configuration

ini

### Provide a log

text

### Code of conduct and contributing guidelines

- [x] Yes, I agree.

I found the difference by doing some FluidSynth builds out of tree straight from upstream. In our `meson.build`, we're setting `enable-floats=true`. There's a CMake setting with the same name upstream. Description is "enable type float instead of double for DSP samples".

We probably want to set this in vcpkg if possible. At least this SC55 soundfont I'm using seems to depend on it.

That's a FluidSynth bug. They had single precision float related bugs in their DSP code before, so looks like there's more:

 URL

Can you report that on the FluidSynth issue tracker as well? I will look into setting 32-bit float mode at build time in the meantime (I think it helps with performance on low-end hardware somewhat).

I'm looking into it. I setup an overlay port for vcpkg to set custom build flags. Then I found that on top of the 32-bit float compile option that needs to be set, there's also a regression somewhere in between 2.3.3 and 2.4.2. So now I'm bisecting FluidSynth.

Bisect done. It was actually the fix for that issue that caused the regression I'm noticing.

First bad commit: URL

First bad version: 2.3.6

All the versions after that commit sound equally as bad with 32-bit and 64-bit.

Nice job :medal_sports: I don't these more recent versions have anything we really need, so we might wanna pin the version to 2.3.5 and ship 0.82.1 with that. I wanna get it out in a few days.

Are you planning on shipping with @shermp 's PR? If not, the Meson build is fine. It's parked on 2.3.3 with the Meson wrap.

Also, I found a vcpkg bug while I was working through this. URL

I put a PR in the ext repo. It uses an overlay port to make changes to the compile flags and downgrade the version.

Not for 0.82.1. But we need the vcpkg fix for the MSVC CI build, so it's great you've done that so quickly 🎖

Nice one, man! 🥳 🚀

Re-opening this as the FluidSynth devs have stated this is not an upstream regression. See comment here: URL

Needs some more investigation. It looks like 2.3.5 and below had a bug that effectively disabled chorus when using `enable-floats=on`. I'm not sure why having no chorus makes this soundfont sound more correct. Is that a thing we should be explicitly disabling or is there another bug in our FluidSynth calling code?

Yeah, I did investigate it a bit just didn't want to change things again before 0.82.1. At least now the behaviour is the same on macOS, Linux and Windows.

I think Trevor's SC-55 SoundFont is just a bit crap 😎 If you check out our FluidSynth integration, a few SF2 files are already blacklisted to auto-disable the chorus with `fsynth_chorus = auto`.

So basically the solution is to just upgrade to the latest FS in 0.83.0, use the single precision path, and put this SC-55 recreation attempt on the blacklist too and call it a day 🤷🏻 We can't fix slightly crappy SF2 files 🤷🏻

I put much more trust in something well-thought-out like GeneralUser GS. The guys seems to know what he's doing, and he's done extensive testing on his SoundFont in a variety of SoundFont players. Trevor's work, on the other hand, seems like a hobbyist effort by a guy who doesn't have too much experience with authoring SF2 files. So I wouldn't use his SF2 as a benchmark for anything.

See the GeneralUser GS author's detailed instructions to get the best out of his SoundFont in DOSBox Staging:
 URL

I'm intending to auto-detect GeneralUser GS in `fsynth_chorus = auto` and `fsynth_reverb = auto` mode and apply his recommended settings in 0.83.0.

Yep, that makes sense. I'll give that a test later with the latest 2.4.4 FluidSynth. Then we can stay up to date with upstream. Hopefully you're right and it's just a buggy SF2 file with a quick and easy workaround via the blacklist.

Yeah, I'm pretty sure about that. I've just commented on the ticket in the FS issue tracker as well:
 URL

We want to be in sync with upstream because a few new interesting improvements are coming, e.g. SC-55-like portamento mode:
 URL

I did a deep-dive into the portamento issue a while ago and concluded FS isn't capable of emulating the SC-55-specific portamento curves, so the best course of action for DOS games it just filter portamento MIDI messages out altogether. With this portamento enhancement, we'll be able to remove that filtering and just set FS into "SC-55 portamento mode".

See here:
 URL

I tested blacklisting the soundfont to disable chorus. Unfortunately, now it sounds incorrect in a different way. Instead of "underwater", it sounds very muffled almost like it's in a tunnel. I tested with 2.3.5 and 2.4.4 with `enable-floats` both on and off and it all sounds wrong.

It appears this soundfont depends on chorus and 2.3.5 was not effectively disabling it as I previously thought. It instead sounds different and this soundfont depends on the old behavior.

We probably should test other soundfonts to see if there's more regressions before upgrading FluidSynth. If this is an outlier, we could print a log message if the user's soundfont contains "SC-55" warning that it's problematic and recommend they try out SoundCanvas instead.

That makes sense to me @weirddan455. With the default settings there was a _lot_ of chorus applied to the sounds. Chorus effects work by adding multiple slightly detuned versions of the signal on top of the original signal. So yeah, with a strong chorus effect you'll get a quite noticeable volume boost!

Consequently, with the chorus removed, we get a big volume drop and the relative volume of the reverb (the tunnel effect you're referring to) thus becomes a lot higher.

This is how Trevor's SF2 sounds compared to NukedSC55 with tweaked FluidSynth chorus and reverb settings. I used FluidSynth 2.4.2 built by the ext project before you introduce the override for this recording. I've just copied the chorus & reverb setting values from the settings' help, then tweaked the level parameter of both until the output started resembling the NukedSC55's output.

recordings.zip

So my conclusion is everthing works as expected. In 0.83.0 we can auto-detect this SF2 file an apply these tweaked settings when `fsynth_chorus` and `fsynth_reverb` is on `auto`. It's probably worth it as it was a popular SF2 prior to NukedSC55.

ini
[midi]
#mididevice = soundcanvas
mididevice = fluidsynth

[fluidsynth]
soundfont = "D:\Emulators\DOS\SoundFonts\SC-55.SoundFont.v1.2b.sf2"
soundfont_volume = 150
fsynth_chorus = 3 0.1 0.3 8.0 sine
fsynth_reverb = 0.61 0.23 0.76 0.15

Have you had a chance to thest this with my tweaked settings @weirddan455 ?

Those settings sound good to me and seem to fix the regression. Tested on 2.3.5 and 2.4.4.

Regression with "enable-floats=on" in DOSBox Staging
### FluidSynth version
2.3.6 is the first affected release. Up to and including 2.4.4 are also affected.
4fc38101f01c5fdc7283054d65c264823b274708 is the first bad commit

### Describe the bug
Playing Doom with an SC-55 soundfont sounds incorrect unless FluidSynth is compiled with "enable-floats=on". Starting with 4fc38101f01c5fdc7283054d65c264823b274708 it sounds incorrect regardless of that setting.

Soundfont: URL
Recordings: URL

The one named `meson.ogg` is the correct sounding one and `vcpkg.ogg` is the incorrect sounding one. It almost sounds like the music is playing underwater.

### Expected behavior
Ideally, it should sound correct regardless of the "enable-floats" setting but I'll settle for fixing the regression and we can continue to use that compile flag.

### Steps to reproduce
Play the first level of Doom in DOSBox staging with a newer version of FluidSynth. Doom should be configured to run with General MIDI. In the .conf file, you should have

[midi]
mididevice = fluidsynth

[fluidsynth]
soundfont = path/to/above/sc55.sf2

As explained in my comment here, the behaviour is all over the place depending on which version you use and whether you do a single or double precision float build:
 URL

Yeah, I get it it's compelling to call a change in behaviour from one FluidSynth minor version to the next a regression, but because of these permutations in behaviour, I'm questioning if there was ever a "correct" behaviour implemented in the first place 😄

Maybe finding answer to a few questions would help: is double precision mode supposed to be the "gold standard"? Have most people been using FluidSynth compiled in double precision mode over the last 5-10+ years? If so, I'm inclined to think that's the "correct" behaviour.

The tricky thing is it's very hard to determine the "correct" behaviour by just comparing the results we get when using some random SoundFonts found on the internet. It doesn't even matter that they're popular; that proves nothing. Some people might have created the an SF2 file with something entirely different than FluidSynth which has slightly different behaviour. Then maybe some SF2 files were authored back in the 90s with Vienna Soundfont Studio for all we know!

I guess SF2 files created with Vienna Soundfont Studio played back on an SB Live! or Audigy board is the "most authentic" standard SF2 playback standard to strive for. But then, if some people authored SF2 files in the past 20 years using FluidSynth or whatever else, those might sound a bit funky on a real SB Live! board... 🥶

So... ultimately I have no idea what is even considered correct here. Please advise 😄

Cheers, so you're saying the double-precision behaviour has always been correct. That's good to know 👍🏻

Yeah, I've heard plenty of modulation effects such as chorus during producing music for about 30 years now 😏

Yep, I've concluded the SF2 file in question is just badly authored, so there's nothing we can do (other than disabling or tweaking the chorus setting to it sounds acceptable).

Yeah I looked into the official specs, and in indeed, it doesn't say anything specific about how the chorus and reverb effects should sound. But probably the default DSP settings on the SB Live! and Audigy boards are a rough "de-facto" standard. Probably your default chorus and reverb settings are close to that.

The author of GeneralUser GS who speaks highly of FluidSynth has started an SF2 compliance suite. There are also real-hardware recordings of the SB Live! and Audigy for reference.

 URL

Yep, I think switching to single-precision would be a good idea. I don't think you're gaining any quality advantage by doing 64-bit float math. Just like you said, you only need to ensure the DSP stuff is stable when using 32-bit floats too.

I am in close contact with Christian here on GH. His test is actually part of our testsuite, although this is unrelated to how chorus / reverb should behave, since there is no definition of "how exactly".