Question: [bug #43577] Extended address not sent with avr109?

I just got an ATxmega32A4U breakout (from MCUZone) and there is no issue to program the flash using AVRISP mkii clone. And the device works fine after the programming (it has two buttons and two LEDs).

PS C:\work\avr\avrdude_test\avrdude_bin> .\avrdude.exe -c avrispmkii -p x32a4u
-U flash:r:atxmega32a4udemo_mcuzone.hex:i

avrdude.exe: AVR device initialized and ready to accept instructions

Reading | ################################################## | 100% 0.01s

avrdude.exe: Device signature = 0x1e9541 (probably x32a4u)
avrdude.exe: NOTE: Programmer supports page erase for Xmega devices.
 Each page will be erased before programming it, but no chip erase is performed.
 To disable page erases, specify the -D option; for a chip-erase, use the -e option.
avrdude.exe: reading flash memory ...

Reading | ################################################## | 100% 0.67s

avrdude.exe: writing output file atxmega32a4udemo_mcuzone.hex

avrdude.exe done. Thank you.

PS C:\work\avr\avrdude_test\avrdude_bin> .\avrdude.exe -c avrispmkii -p x32a4u -e

avrdude.exe: AVR device initialized and ready to accept instructions

Reading | ################################################## | 100% 0.01s

avrdude.exe: Device signature = 0x1e9541 (probably x32a4u)
avrdude.exe: erasing chip

avrdude.exe done. Thank you.

PS C:\work\avr\avrdude_test\avrdude_bin> .\avrdude.exe -c avrispmkii -p x32a4u
 -U flash:w:atxmega32a4udemo_mcuzone.hex:i

avrdude.exe: AVR device initialized and ready to accept instructions

Reading | ################################################## | 100% 0.01s

avrdude.exe: Device signature = 0x1e9541 (probably x32a4u)
avrdude.exe: NOTE: Programmer supports page erase for Xmega devices.
 Each page will be erased before programming it, but no chip erase is performed.
 To disable page erases, specify the -D option; for a chip-erase, use the -e option.
avrdude.exe: reading input file atxmega32a4udemo_mcuzone.hex for flash
avrdude.exe: writing 36786 bytes flash ...

Writing | ################################################## | 100% 3.00s

avrdude.exe: 36786 bytes of flash written
avrdude.exe: verifying flash memory against atxmega32a4udemo_mcuzone.hex

Reading | ################################################## | 100% 0.67s

avrdude.exe: 36786 bytes of flash verified

avrdude.exe done. Thank you.

[bug #43577] Extended address not sent with avr109
Arne Rossius
Tue 11 Nov 2014 02:14:43 PM UTC

When programming the ATxmega256a3 with avrdude 6.1 (windows version), the extended address is not sent when jumping across a 128 KiB boundary. I have created a text hexfile which only contains data for flash pages 0 (word address 0x00000) and 511 (word address 0x1FF00). However as can be seen from the attached log, after programming the first page only a 16-bit address command ('A') is sent instead of the 24-bit address command ('H'), resulting in an address of 0x0FF00 instead of 0x1FF00. Verify doesn't fail because the same happens when reading, but does fail when programming an actual application (without a gap) larger than 128 KiB.

The programmer is the "xboot" bootloader < URL The avrdude.conf is unchanged.

Programming the attached hex file does work with the older avrdude 5.11 because this version programs pages 1 to 510 with 0xFF, only sending the address once before the first page and then relying on the bootloader's auto-increment to reach the high addresses. As can be expected, it takes a very long time though.

Also, as can be seen from the xboot source, it resets the extended address byte to zero when receiving an 'A' command. I'm not sure if this behaviour is correct, but if it is, this means that avrdude must send an 'H' command for all flash pages above 128 KiB (unless no address is sent at all between adjacent pages).

file #32418: napftest.hex
file #32419: avrdude-error.txt

This issue was migrated from URL

Ref: xmega relared issues
* URL
* URL
* URL
* URL
* URL
* URL
* URL
* URL
* URL
* URL
* URL
* URL
* URL (but also affect other AVR chips).
* URL
* URL
* URL

My test results in #454 show that the writing is good but not verification. But that is based on ATmega2560 with xboot, not ATxmega256a3. So ATxmega256a3 may get more issues (need to get some one to help testing as I do not have ATxmega256a3).

* URL

@MCUdude

Now that you have access to the ATxmega256A3BU Xplained board, please take a look at this one and #404 when you got a chance. Thanks.

I compiled xboot for ATxmega256A3BU, and I'm able to communicate with Avrdude using the bootloader:

$ ./avrdude -cavr109 -p atxmega256a3bu -b 115200 -P /dev/cu.usbserial-1410

Connecting to programmer: .
Found programmer: Id = "XBoot++"; type = S
 Software Version = 1.7; No Hardware Version given.
Programmer supports auto addr increment.
Programmer supports buffered memory access with buffersize=512 bytes.

Programmer supports the following devices:
 Device code: 0x7b

avrdude: AVR device initialized and ready to accept instructions

Reading | ################################################## | 100% 0.00s

avrdude: Device signature = 0x1e9843 (probably x256a3bu)

avrdude done. Thank you.

I can also read from flash:

$ ./avrdude -cavr109 -p atxmega256a3bu -b 115200 -P /dev/cu.usbserial-1410 -Uflash:r:-:I

Connecting to programmer: .
Found programmer: Id = "XBoot++"; type = S
 Software Version = 1.7; No Hardware Version given.
Programmer supports auto addr increment.
Programmer supports buffered memory access with buffersize=512 bytes.

Programmer supports the following devices:
 Device code: 0x7b

avrdude: AVR device initialized and ready to accept instructions

Reading | ################################################## | 100% 0.00s

avrdude: Device signature = 0x1e9843 (probably x256a3bu)
avrdude: NOTE: Programmer supports page erase for Xmega devices.
 Each page will be erased before programming it, but no chip erase is performed.
 To disable page erases, specify the -D option; for a chip-erase, use the -e option.
avrdude: reading flash memory ...

Reading | ################################################## | 100% 26.46s

avrdude: flash is empty, resulting file has no contents
avrdude: writing output file
:00000001FF

avrdude done. Thank you.

However, I'm not able to write:

$ ./avrdude -cavr109 -p atxmega256a3bu -b 115200 -P /dev/cu.usbserial-1410 -Uflash:w:/Users/hans/Downloads/XMEGA_A3BU_XPLAINED_DEMO1.hex:i

Connecting to programmer: .
Found programmer: Id = "XBoot++"; type = S
 Software Version = 1.7; No Hardware Version given.
Programmer supports auto addr increment.
Programmer supports buffered memory access with buffersize=512 bytes.

Programmer supports the following devices:
 Device code: 0x7b

avrdude: AVR device initialized and ready to accept instructions

Reading | ################################################## | 100% 0.00s

avrdude: Device signature = 0x1e9843 (probably x256a3bu)
avrdude: NOTE: Programmer supports page erase for Xmega devices.
 Each page will be erased before programming it, but no chip erase is performed.
 To disable page erases, specify the -D option; for a chip-erase, use the -e option.
avrdude: reading input file /Users/hans/Downloads/XMEGA_A3BU_XPLAINED_DEMO1.hex for flash
avrdude: writing 35062 bytes flash ...

Writing | # | 1% 0.00s ***failed;
 ***failed;
 ***failed;
 ***failed;
 ***failed;
 ***failed;
 ***failed;
 ***failed;
 ***failed;
 ***failed;
 ***failed;
 ***failed;
 ***failed;
 ***failed;

...

 ***failed;
 ***failed;
 ***failed;
 ***failed;
 ***failed;
 ***failed;
 ***failed;
 ***failed;
 ***failed;
Writing | ################################################## | 100% 0.10s

avrdude: 35062 bytes of flash written
avrdude: verifying flash memory against /Users/hans/Downloads/XMEGA_A3BU_XPLAINED_DEMO1.hex

Reading | ################################################## | 100% 3.46s

avrdude: verification error, first mismatch at byte 0x0000
 0xff != 0xec
avrdude: verification error; content mismatch

avrdude done. Thank you.

It turns out that I forgot to add the `-e` flag. Now I can get the bootloader to work. I'll try with a bigger file soon.

$ ./avrdude -cavr109 -p atxmega256a3bu -b 115200 -P /dev/cu.usbserial-1410 -e -Uflash:w:/Users/hans/Downloads/XMEGA_A3BU_XPLAINED_DEMO1.hex:i

Connecting to programmer: .
Found programmer: Id = "XBoot++"; type = S
 Software Version = 1.7; No Hardware Version given.
Programmer supports auto addr increment.
Programmer supports buffered memory access with buffersize=512 bytes.

Programmer supports the following devices:
 Device code: 0x7b

avrdude: AVR device initialized and ready to accept instructions

Reading | ################################################## | 100% 0.00s

avrdude: Device signature = 0x1e9843 (probably x256a3bu)
avrdude: erasing chip
avrdude: reading input file /Users/hans/Downloads/XMEGA_A3BU_XPLAINED_DEMO1.hex for flash
avrdude: writing 35062 bytes flash ...

Writing | ################################################## | 100% 4.29s

avrdude: 35062 bytes of flash written
avrdude: verifying flash memory against /Users/hans/Downloads/XMEGA_A3BU_XPLAINED_DEMO1.hex

Reading | ################################################## | 100% 3.46s

avrdude: 35062 bytes of flash verified

avrdude done. Thank you.

@mcuee I can confirm that there's an issue writing to addresses above 128kiB.

$ ./avrdude -cavr109 -p atxmega256a3bu -b 115200 -P /dev/cu.usbserial-1410 -e -Uflash:w:0x55_256kib.hex

Connecting to programmer: .
Found programmer: Id = "XBoot++"; type = S
 Software Version = 1.7; No Hardware Version given.
Programmer supports auto addr increment.
Programmer supports buffered memory access with buffersize=512 bytes.

Programmer supports the following devices:
 Device code: 0x7b

avrdude: AVR device initialized and ready to accept instructions

Reading | ################################################## | 100% 0.00s

avrdude: Device signature = 0x1e9843 (probably x256a3bu)
avrdude: erasing chip
avrdude: reading input file 0x55_256kib.hex for flash
avrdude: writing 262143 bytes flash ...

Writing | ################################################## | 100% 31.83s

avrdude: 262143 bytes of flash written
avrdude: verifying flash memory against 0x55_256kib.hex

Reading | ################################################## | 100% 25.66s

avrdude: verification error, first mismatch at byte 0x1ffff
 0xff != 0x55
avrdude: verification error; content mismatch

avrdude done. Thank you.

[bug #39033] Support for ATxmegaXXXA4U devices
Brian Mayton
Tue 21 May 2013 06:30:25 PM UTC

avrdude appears to be able to communicate with an ATxmega128A4U in PDI mode, but an attempt to program flash memory fails at verification; the memory appears to read back as 0xFF.

It looks like avrdude can read the flash correctly; if I program the chip using Atmel Studio, a quick glance at 'dump flash' in avrdude's terminal mode looks like an intact interrupt table.  A chip erase from avrdude also appears to work; the flash reads back as 0xff after executing avrdude -e.

I captured the attached traces from Atmel Studio successfully programming (and then reading back) the flash memory.  The hex file used is the appropriate xmega DFU bootloader image from AVR1916.  The files were captured with Atmel Studio 6 running in VMware and wireshark on Linux capturing with the usbmon facility.  I would be happy to capture additional operations if it would be useful.

file #28129: atxmega128a4u_dumps.tar.bz2

This issue was migrated from URL

Joerg Wunsch
Fri 13 Sep 2013 02:16:15 PM UTC

After someone just got me an Xmega-A3BU Xplained board, I
gave it a try, and had no troubles (beyond the rather
complicated board setup) to talk PDI to it, using a JTAGICE3.

@dl8dtl There are many xmega related issues, not so sure this one (as well as others) is still relevant or not.

I have tried to link the xmega related issues here:
* URL


Question: Nil response?

I would recommend using bundles for policy and data distribution. Here's an example tutorial.

Nil response
We are using OPA in a k8s environment.
* OPA version: 0.56.0
* Response is nil
* flags
* "run",
 "--server",
 "--config-file=/config/config.yaml",
 "--log-level=info",
 "--log-format=json",
 "--authentication=token",
 "--authorization=basic",
 "/authz/authz.rego",
 "/authz/bearer.rego",
 "--ignore=.*",
 "--addr= URL

## Steps To Reproduce
The error happen over time. No discernible pattern other than a restart of OPA fixes it.

When the "nil" response starts happening it is reproducible until OPA is restarted.
Is there a way to gather more information when this happens?

@kirk-patton closing this for now. Feel free to re-open with more details about the issue and steps to reproduce it. Thanks!

@ashutosh-narkar the details were hidden as they had been provided as part of the comment included in the template. I edited the issue to unhide that.

Looks like this was resolved in URL please re-open if that isn't the case.

OPA response is nil issue
@kirk-patton I would recommend using the latest version of OPA and try to repro with that. Also it would help if you could share the policy, input etc. There were some issues around OPA returning a "nil" response in an older version which we fixed. Not sure if you're seeing the same issue but using the latest OPA version would help to figure that out.

_Originally posted by @ashutosh-narkar in URL

I have updated the clusters to opa 0.66.0. Things were quite for a few weeks. The issue has resurfaced though. I tried asking in the community slack channel, but so far no response.

The decision logger we use does not return the admission review when the response from OPA is nil. When the response is nil, should be possible to get the admissionReview? I would like to be able to provide more specifics, but I am having trouble getting any information at the time of the issue. I do know that doing a rollout restart of the opa deployment alleviates the issue for a time hours/a couple days. The issue seems more prevalent in certain clusters.

If you could provide some details that help repro the issue that would be really helpful.

If only I could reproduce the issue :-)

We use OPA as and admission webhook in multiple k8s clusters. We load our policies using kube-mgmt. We have a decision logger configured to log OPA policy success/fail with some details to aid in troubleshooting. If OPA returns a true/false response, we can see the k8s admissionReview just fine. But, if OPA returns a nil response, the admissionReview is also nil.

Given that the decision logger does not return any useful information when the response from OPA is nil, there is not much to go on.

Does OPA expose the k8s admission review details when OPA response is nil? Or is there some other recommended way to get useful information to help

From our decision logger..

type DiagnosticsResponseElementV1 struct {
 Revision string `json:"revision,omitempty"`
 DecisionID string `json:"decision_id,omitempty"`
 RemoteAddr string `json:"remote_addr"`
 Query string `json:"query"`
 Path string `json:"path"`
 Timestamp string `json:"timestamp"`
 Input v1beta1.AdmissionReview `json:"input,omitempty"`
 Result *v1beta1.AdmissionReview `json:"result,omitempty"`. <====
 Error *types.ErrorV1 `json:"error,omitempty"`
 Explanation types.TraceV1 `json:"explanation,omitempty"`
 Metrics types.MetricsV1 `json:"metrics,omitempty"`
}

We are not running into any issues when we run the unit tests on our policies. We alert when OPA returns a nil response, as it impacts the users submitting to the cluster. Performing a k8s rolling restart of the OPA deployment temporarily resolves the issue. Jobs submitted that encounter the `OPA response is nil`, submit fine after OPA is restarted.

I have only looked at the decision-logger side-car when the issue has surfaced. I will make a point to examine OPA logs to see if there are any clues there.

Thank you for confirming that! I will update the decision logger to spew.Dump to the console and see what that gets me

I am working on gathering more information to help diagnose this issue.

Looks like the issue we are seeing is the same as URL

OPA is getting restarted on our AWS cluster, possible dues to Karpenter. kube-mgmt does not reload the policies. The response we get back just contains the decision-id and nothing else.

We are planning on adding a livenessProbe to the `mgmt` side-car


Question: 选项卡菜单item名错误?

Babel 2022-12 is still not released (link above contains only 2021-12 version).
However we can see it in update sites: URL

The roblem is that we can't use update site, we need archived zip with all translations. I think we need to contct Babel team and ask to publich the release.

选项卡菜单item名错误
### Description

image

### DBeaver Version

Community 22.1.5.202208211008

### Operating System

Windows11

### Database and driver

_No response_

### Steps to reproduce

_No response_

### Additional context

_No response_

Hello @shawnWXN,

Please use English and provide more information about the issue.

If the localization is incorrect, then please suggest a better translation.

Closed as duplicate of URL

This is the bug from the Eclipse site

Update Babel version
### Description

<img width="249" alt="image" src=" URL

### DBeaver Version

22.2.0.202209051344

### Operating System

_No response_

### Database and driver

_No response_

### Steps to reproduce

_No response_

### Additional context

_No response_

Hello @lastares

This is a translating issue from the Eclipse side: URL

Probably, we need to find or create a ticket on the Eclipse bug tracker.

We probably need to update Eclipse Babel from our side first.

We need to wait for the new babel build to have correct translation.

OK

蓝笑灵晨
***@***.***

------------------ 原始邮件 ------------------
发件人: "dbeaver/dbeaver" ***@***.***>;
发送时间: 2023年1月17日(星期二) 晚上7:59
***@***.***>;
***@***.******@***.***>;
主题: Re: [dbeaver/dbeaver] close tab repeated (Issue #17697)

The latest build contains correct translation.

—
Reply to this email directly, view it on GitHub, or unsubscribe.
You are receiving this because you were mentioned.Message ID: ***@***.***>

I found a temporary solution to this problem.

Taking the latest version 23.2.2 as an example, find following jar file:

, and use decompression software to extract the Chinese localization attribute file from it:

, and then change the value corresponding to the key `menuCloseAll` to `\u5173\u95ed\u5168\u90e8(&A)`,

Then, replace the original file in the Jar package with the modified file.

<img width="743" alt="2023-10-10_145621" src=" URL

So I used the latest language pack from official update site (pls see the screenshot),
image

the unpacked .jar already contains required fix
image


Question: Performance issue on IntelliSense after the verion 1.18.5?

@Colengms @LittleFatHero The performance issue started with 1.19.x -- it looks like it may be "by design" due to the workspace symbol search changes. I'm seeing roughly it take roughly twice the amount of time to parse/update the database and the database is roughly 50% bigger. We'll look into whether that can be improved.

Performance issue on IntelliSense after the verion 1.18.5
### Environment

- OS and Version:
Local mode: mac
Remote mode: mac-client, linux-server
- VS Code Version: 1.18.9 dc96b837cf6bb4af9cd736aa3af08cf8279f7685
- C/C++ Extension Version: after 1.18.5
- If using SSH remote, specify OS of remote machine: centos 7.9 (or on mac locally)

### Bug Summary and Steps to Reproduce

Bug Summary:
IntelliSense is rather slowly, prevent the functions which are dependent on IntelliSense. Whatever file I open, the IntelliSense process is spining for even a minute, and before it finished I can do nothing about the functions which are dependent on the IntelliSense.

Steps to reproduce:
1. Run vscode on mac locally or a linux server remotely, checkout the project: URL master branch is ok.
2. install the c++ extension pack (which contains cpptools of version greatter than 1.18.5)
3. Open a file under the forder $project-root/be/sre.
4. Try the functions releated with IntelliSense.

Expected behavior:
IntelliSense is extremlly slow, and prevent you using the functions which are denpendent on the IntelliSense.

### Configuration and Logs

shell
/

### Other Extensions

_No response_

### Additional context

I try the cpptools extension version by version, and I found that the IntelliSense is smooth at (and before) the version 1.18.5, if I update the extesion to version which is greater than 1.18.5, the issue will appear.

So it may be helpful that checkout the changes between 1.18.5 and 1.19.5.

@XuPengfei-1020 Can you set C_Cpp.loggingLevel to "Debug" and look at the C/C++ logging after opening a file logging text that indicates what is blocking the IntelliSense operation? In particulary, what does it show for "Update IntelliSense time (sec):" or mentions of a filename cache.

I have tried clangd extension, I am sorry to say that there has a long way for cpptools extension to go.
Clangd is far more smooth than cpptolls(even 1.17.5 version) on IntelliSense, just for now (may be caused by the project I opend is a bit large, it contains more than 3k cpp/h files).

Now I have to use clangd extension.
I will switch to cpptools later after this bug has been fixed.
Thank you.

LSP: Sending response (id: 158)
LSP: Sending response (id: 154)
LSP: Sending response (id: 157)
**Update IntelliSense time (sec): 26.648**
LSP: (received) cpptools/getFoldingRanges: file:///root/palo/be/src/olap/cumulative_compaction_time_series_policy.h (id: 159)
LSP: (invoked) cpptools/getFoldingRanges: file:///root/palo/be/src/olap/cumulative_compaction_time_series_policy.h (id: 159)

Hi @XuPengfei-1020 , @heartacker . Can you be more specific about the version number that introduced the issue, by trying out the available 'insiders' versions between 1.18.5 and 1.19.5?

This number indicates when IntelliSense processing has gone idle for all open files. This will have increased in 1.19.2 due to IntelliSense improvements to update files 'progressively'. IntelliSense will be functional, and the currently visible portion of the file(s) on screen will be updated, colorized, etc., _much_ more quickly than previously. This number is not a representation of how quickly IntelliSense is usable or results are available for any particular file. This number is not an indicator of IntelliSense performance. IntelliSense will be fully functional before this is logged.

If you enable `"C_Cpp.loggingLevel": "7"`, you can see individual progressive updates. As soon as you see the first such update, the visible portion of the file should be fully updated and IntelliSense ready to use. The remaining updates, up until `Update IntelliSense time` indicate work to populate colorization, errors, etc., outside of the currently visible range of the file, asynchronously.

Opening the doris project at the suggested path, with that log level set, and no other configuration (as none was mentioned in the issue), IntelliSense does not appear to be configured for me. It's necessary to configure the C/C++ Extension with the same information used to build the code, for the same reason that it's needed to build the code - IntelliSense is a front-end C++ compiler. It needs to know the defines and include paths necessary to parse/build the code. Without this, the C/C++ Extension may perform an excessive amount of work trying to access files it's unable to find and try to parse code it's not been configured to parse correctly. When using a large project without a configuration, I would not expect most IntelliSense functionality to work properly or quickly.

I don't think we yet have enough information to investigate this issue. Could either of you please provide concise repro steps with a project that is configured for IntelliSense? The doris project does appear to include CMakeLists.txt files, so the C/C++ Extension could potentially be configured using the CMake Tools extension. But this did not appear to get configured for me 'out of the box', given the complex directory structure of that project.

I'm unable to reproduce any obvious performance issue with files in the doris project. To investigate further, I think we'll need more information, such as an example project and an associated correct IntelliSense configuration, specific files to open, and other specific steps on what operations to perform and what is not as expected. If we're not able to repro locally, we may also need detailed logs from your own repro(s). How to collect that information is in our new issue template, and would generally include the results of the `C/C++: Log Diagnostics` command (with the reproing file open), setting and configuration details, and C/C++ output logs with `C_Cpp.loggingLevel` set to `Debug`. The C/C++ Extension may function very different depending on how/if it's been configured and what features are used, so we really do need very specific information to investigate repros.

名称: C/C++
ID: ms-vscode.cpptools
说明: C/C++ IntelliSense, debugging, and code browsing.
版本: 1.20.5 @Colengms
发布者: Microsoft
VS Marketplace 链接: URL

"This number is not an indicator of IntelliSense performance. IntelliSense will be fully functional before this is logged."
@Colengms Very appreciate for your patient explanation.

Does the IntelliSence dependent the cmake-tools extension?
After I config the cmake-tools and run it, I got a warn msg: , see:
”Using 'DEBUG' for CMAKE_BUILD_TYPE instead of 'Debug' to ensure that IntelliSense configurations can be found.“
I don`t know whether the IntelliSense issue is related with this.
Details about it in this issue I posted to cmake-took extension: URL

Env of project Doris and the step of producing the issue:
1. Clone the doris project from URL checkout the tag: [2.1.0-release]
2. Pull the doris dev docker image: apache/doris:build-env-ldb-toolchain-latest, either linux or windows docker-desktop is ok.
3. Run the docker image, mount the doris project root folder on the ~/doris in container (Or pull the doris project in containers, anyway is ok).
4. Start vscode(latest version is ok), connect into the container(either by ssh or local-container-extension is ok), run the vscode as remote mode.
5. Install the c++-extension-pack(latest version is ok), and config the cmake-tools (compiler is /var/local/ldb-toolchain/bin/clang, src is $doris/be/).
6. Try to open some files under the folder $project/be/src/olap/(such as tablet.cpp, table.h)
7. Try to go to the definition of the included files using cmd/ctrl + left-click on .
8. Try to go to the definition of the functions/symbols which defined in other cpp/h file and referenced by current file.

I think this is a duplicate of URL and a regression caused by adding more stuff to the database for 1.19.x. This might get fixed/improved by not blocking IntelliSense on tag parsing with 1.21.2+.

IntelliSense Updating and Parsing Workspace speed is much slower after 1.18.5
### Environment

- OS and Version: windows 10
- VS Code Version: 1.89.0
- C/C++ Extension Version: 1.20.5
- If using SSH remote, specify OS of remote machine: ubuntu 20.04

### Bug Summary and Steps to Reproduce

Bug Summary:
My vscode is running in windows 10 and I connected to remote machine ubuntu 20.04 which has installed vscode server and cpptools using the ssh remote-SSH extensions. when I open a C source file and go to definition of a function， it could not jump to the definition because cpptools is Updating IntelliSense and Parsing Workspace. So I wait for cpptools to finish the job, but I found that it Parsing Workspace is extremely slower than previous version. My workSpace has about 100000 C source files.
1. Why cpptools 1.20.5 verison feels slow than previous version?
2. what is strategy of when Updating IntelliSense and Parsing Workspace? will it updating of the file on the workspace or only update the and parse the open file?
3. Would it be possible to add a setting that allows the user to specify number of thread to use for parsing?

Steps to reproduce:
1. Open the workspce which has about 100000 C source Files using vscode 1.89.0 and cpptools 1.20.5
3. Count the time it takes to finish update Intellisense and Parsing workspace.
4. compare the time with privous version of vscode and cpptools

Expected behavior:
update Intellisense and Parsing workspace faster

### Configuration and Logs

shell
default config

### Other Extensions

_No response_

### Additional context

_No response_

Hi @LittleFatHero . Would you be able to try out the various versions (including prerelease versions) between 1.89 and 1.20.5, to isolate which version seems to introduce the performance regression?

Hi @Colengms , I'm sorry, I have been busy doing other thing and has no time to do this action. When the cpptools release a new version , is there any performance test about parse and intellisense? if not , maybe the team can do this when a new version is going to release. When there is a performance regression, the new version should not release?


Question: Media location for a folder per post?

Doesn't mean I won't get something sorted out ultimately. I just need time to sit down and figure out how I would handle it.

The main issue with this Hugo flow is that you need to define the name of the folder (and thus the slug) before you upload the assets. But in Pages CMS, for new entries, the slug/filename is set up on saving the entry.

If you have suggestions, let me know.

Media location for a folder per post
What's the configuration for a setup like this? All what I have tried so far didn't work and keeps assuming I have one folder for ALL my media, which is very chaotic.

Folder structure is simply like this

content/
 posts/
 post1/
 index.md
 image1.png
 post2/
 index.md
 image1.png
 post3/
 index.md
 image1.png

I second this.

I really prefer keeping assets (images mostly, but sometimes other file types) in the same place as the text content using them.

I think this is related to what's been asked in another issue:
- #14

I think to enable this I would probably need to do 2 things:

1. Enable saving posts in a subfolder, meaning that rather than saving an entry as `posts/my-entry.md` I would save it as `posts/my-entry/index.md`.
2. For new content, force the user to save the entry before uploading any asset.

The issue with the first point is that this will likely require a lot more work as to retrieve all the entries in a collection I'll need to run an individual query to the GitHub API for each entry as I don't think I can recursively fetch files with the GitHub GraphQL API. To make it manageable, I'll likely need to aggressively cache entries. That's probably more of a 1.0.0 version to be honest.

@itstamimi @nhoizey how do you think the approach would work with media choosers or re-used images?

I sometimes re-use images on multiple pages, I may have list and detail pages that re-use imagery although often at different sizes. Some times I just want to open a media browser and pick an image to use. It's easy with a git backend to list a folder and a few children, it's harder when the media browser has to look all over the repo to build it's collection.

Discussed in other threads. I'll keep this in mind in the future, but others seem to have been able to make their Astro or Hugo setup work without this (e.g. #129).

How to set up featured-image paths in Hugo
Hello! My Hugo blog files are all in individual folders under `content/posts`, and the blog images are in the same folder as the Markdown files. I have already solved the issue of uploading and referencing images in the body text, but the automatically generated featured image payh cannot be rendered by Hugo due to incorrect paths. I am unable to resolve this path issue by modifying the Hugo template because there are already hundreds of articles using the original method.

## Pages CMS setting

media:
 input: content/posts
 output: /content/posts
content:
 ...
 - name: image
 label: Image
 type: image
 list: true
 options:
 input: content/posts
 output: /content/posts

## Hugo Blog Needed

---
title: "test"
image:
 - 1.webp # featured-image
---

## Pages CMS Output Now

---
title: "test"
image:
 - content/editor/test/1.webp
---

It's been asked a few times for this (for Hugo specifically).

I've not yet figured out a way to handle this gracefully in Pages CMS. The expectation right now is that you upload all media in the same folders, not individual folders for your content.

Thank you, I'm completely heartbroken. I'll just keep using some IDE tools like VSCode, but it's really inconvenient on a phone.

After hearing you describe the issue, I actually came up with a solution. I set up a dedicated area for adding images, which uploads images to the same directory as the Markdown file, and then set the featured image as a string, manually filling in the image address. I didn't expect it to be so simple.

That's great!

media:
 input: content/post
 output: /content/post
 categories: [image]
content:
 - name: posts
 label: Posts
 type: collection
 path: content/post
 filename: index.md
 view:
 fields: [ title, date, draft ]
 fields:
 - { name: title, label: Title, type: string, required: true }
 - { name: draft, label: Draft, type: boolean, default: true }
 - { name: slug, label: Slug, type: string, required: true }
 - name: date
 label: Date
 type: date
 options:
 format: yyyy-MM-dd'T'HH:mm:ss
 time: true
 - name: categories
 label: Categories
 type: select
 list: true
 options:
 values:
 - legal
 - cases
 - name: tags
 label: Tags
 type: string
 list:
 min: 1
 max: 5
 - name: uploadimage
 label: Uploadimage
 type: object
 list: true
 fields:
 - name: newimage
 label: newimage
 type: image
 options:
 input: content/post
 output: content/post
 categories: [image]
 - { name: image, label: image, type: string, required: true }
 - name: body
 label: Body
 type: rich-text

Great to hear that. I'll close this issue for now, I'll still have a look at supporting a more complex setup for Hugo if possible.

I believe I have fully resolved the issue with the Hugo directory structure.

There are three settings that need to be adjusted:

 1. In the Pages CMS, do not fill in the media output path with "content"; just keep it as "posts".

2. Set Hugo's permalink configuration to the default format, which uses the name of the folder containing index.md as the slug.

The purpose of these two steps is to ensure that the image URLs in Pages CMS and the default Hugo format of a both output the same URL after Hugo's rendering.

3. Customize an image hook in Hugo to extract only the filename part from the automatically generated a link in Pages CMS and render the file.

This is done so that Hugo treats a and a as the same element for rendering.

Ultimately, the problem has been successfully resolved.


Question: VMs don't boot on 4Kn drive?

Sector size is advertised by the block backend in xenstore (`xenstore-ls /local/domain/0/backend/vbd/$DOMID/51712`), but I don't see any option to force specific value.

This issue is really unfortunate, because a lot of places in Qubes assume you can freely transfer disk image and it will work just fine. This include cloning VMs (including cloning to a different storage pool), backup/restore etc. So, the solution here should be either:
- find a way to construct a disk image to work on both 4k and 512 sector size
- force VM to see 512 sector size

The second one may come with a performance penalty. The first one would not have this problem, but not sure if it's possible. I'm fine with making partition table 4K aligned, as long as it will also work with 512 sector size. But it isn't clear to me it would be enough.

Partition table and filesystem are built here: URL

Another idea would be to revert to a filesystem directly on /dev/xvda (without any partition table). This may not be as simple as it sounds, because we need to fit grub somewhere (with HVM with in-VM kernel case).

But this all may not work for other cases, including other OS. Imagine installing some OS (Linux, Windows, whatever) in a standalone HVM and then moving it to another storage pool (or restoring a backup on another machine). Those cases may require emulating constant sector size.

Sadly, I don't have any hardware with 4k physical sector size to test on. I'll try to find a way to emulate one.

BTW, another issue from 4k sector size is 8GB of swap, instead of 1GB. But this should be easy to fix in this script

VMs don't boot on 4Kn drive
How to file a helpful issue

### Qubes OS release

R4.1.1 **TESTING**

### Brief summary

Using default "LVM" installation layout.
When I install Qubes with "LVM" disk layout, the default pool is varlibqubes.
The VMs are all based on stock templates that are shipped with the iso. I didn't build 4Kn templates.
All my VMs don't boot after I upgrade dom0 to tesing-latest.
Without enabling testing, things are fine.

### Steps to reproduce

0. Find a NVMe drive using LBA sector size 4096.
1. Install Qubes using "LVM" disk layout, and upgrade Qubes to latest (xen 4.14.5-7).
2. Find that VMs do boot.
3. Upgrade Qubes to testing-latest (without kernel latest) (xen 4.14.5-9).
4. Find that no VM wants to boot.

### Expected behavior

VMs boot as usual.

### Actual behavior

An error message, asserting "Start failed: internal error: libxenlight failed to create new domain xxx"

Here are the logs in /var/log/libvirt/libxl/libxl-driver.log:

2022-10-20 12:23:19.476+0000: libxl: libxl_exec.c:117:libxl_report_child_exitstatus: /etc/xen/scripts/block add [5543] exited with error status 1
2022-10-20 12:23:19.476+0000: libxl: libxl_device.c:1302:device_hotplug_child_death_cb: script: losetup --direct-io=on /dev/loop8 /var/lib/qubes/appvms/sys-net/private-dirty.img failed
2022-10-20 12:23:19.570+0000: libxl: libxl_exec.c:117:libxl_report_child_exitstatus: /etc/xen/scripts/block add [5542] exited with error status 1
2022-10-20 12:23:19.570+0000: libxl: libxl_device.c:1302:device_hotplug_child_death_cb: script: losetup --direct-io=on /dev/loop9 /var/lib/qubes/appvms/sys-net/root-dirty.img failed
2022-10-20 12:23:19.666+0000: libxl: libxl_exec.c:117:libxl_report_child_exitstatus: /etc/xen/scripts/block add [5548] exited with error status 1
2022-10-20 12:23:19.666+0000: libxl: libxl_device.c:1302:device_hotplug_child_death_cb: script: losetup --direct-io=on -r /dev/loop10 /var/lib/qubes/vm-kernels/5.15.74-1.fc32/modules.img failed
2022-10-20 12:23:19.775+0000: libxl: libxl_exec.c:117:libxl_report_child_exitstatus: /etc/xen/scripts/block add [5545] exited with error status 1
2022-10-20 12:23:19.775+0000: libxl: libxl_device.c:1302:device_hotplug_child_death_cb: script: losetup --direct-io=on /dev/loop11 /var/lib/qubes/appvms/sys-net/volatile-dirty.img failed
2022-10-20 12:23:19.775+0000: libxl: libxl_create.c:1686:domcreate_launch_dm: Domain 3:unable to add disk devices
2022-10-20 12:23:19.828+0000: libxl: libxl_exec.c:117:libxl_report_child_exitstatus: /etc/xen/scripts/block remove [5805] exited with error status 1
2022-10-20 12:23:19.829+0000: libxl: libxl_device.c:1302:device_hotplug_child_death_cb: script: /etc/xen/scripts/block failed; error detected.
2022-10-20 12:23:19.839+0000: libxl: libxl_exec.c:117:libxl_report_child_exitstatus: /etc/xen/scripts/block remove [5804] exited with error status 1
2022-10-20 12:23:19.839+0000: libxl: libxl_device.c:1302:device_hotplug_child_death_cb: script: /etc/xen/scripts/block failed; error detected.
2022-10-20 12:23:19.849+0000: libxl: libxl_exec.c:117:libxl_report_child_exitstatus: /etc/xen/scripts/block remove [5807] exited with error status 1
2022-10-20 12:23:19.849+0000: libxl: libxl_device.c:1302:device_hotplug_child_death_cb: script: /etc/xen/scripts/block failed; error detected.
2022-10-20 12:23:19.858+0000: libxl: libxl_exec.c:117:libxl_report_child_exitstatus: /etc/xen/scripts/block remove [5810] exited with error status 1
2022-10-20 12:23:19.859+0000: libxl: libxl_device.c:1302:device_hotplug_child_death_cb: script: /etc/xen/scripts/block failed; error detected.

The fix will be in repo soon. In the meantime, you can recover by editing `/etc/xen/scripts/block` and removing `--direct-io=on` parameter from `losetup` call.

Whoops! I had not considered that at all. Glad this got caught in testing!

This is fixed in util-linux v2.38+, although we'd also have to pass an explicit `--sector-size=512` argument to ensure that the incompatible case is automatically resolved in favor of turning direct I/O off (as opposed to changing the logical block size).

@SurFlurer did you by any chance manually set up a 4k dm-crypt sector size? The old `cryptsetup` in the R4.1.x installer defaults to 512 byte sectors even on drives with 4k logical sectors, so normally your LVM Thin pool should have arrived at 512 byte sectors as well and avoided this bug.

@rustybird

In fact, I'm using LVM thick pool, because if I choose to use the LVM thin layout in the installer, there will be an error message and no new partition will be created.

Whoops, I forgot to explicitly emphasize that "LVM" is different from LVM-thin.

However, if my memory is correct, the default pool in LVM thin layout is “vm”.

Oh man, I always assumed that dm-crypt with sector_size 512 (the default on R4.1.x) would "shield" the upper storage layers from the 4Kn drive's logical block size. But actually it only works the other way around, which is sector_size 4096 bumping up a physical or logical block size of 512 on the underlying device to 4096 on the dm device.

That explains why this bug was able to affect your system.

General summary: People with 4Kn drives currently can't use lvm_thin pools (#4974), unless they're doing esoteric things like building custom template image files etc. And people who set up 4K dm-crypt on 512e drives have the same problem. But installation layouts compatible with the file-reflink driver are okay: "Btrfs", "Standard Partition", or "LVM" that's not "Thin". The latter two use XFS for the varlibqubes pool and are somewhat less tested, as you noticed :wink:

Support 4k storage
**Qubes OS version**

R4.0

**Affected component(s) or functionality**

VMs not working/starting right from a fresh install.

**Brief summary**

Right after a fresh install, all VMs fail to mount root and therefore fails to start beyond the point where they expect /dev/xvda3 available. This happens on a device that has 4kB logical and physical block sizes (NVMe drive). This was not problem in R3.2 (as it used files by default for VM storage).

**To Reproduce**

Steps to reproduce the behavior:
1. Install Qubes to a drive with 4kB sector size (both logical / physical); (I put /boot to a SATA drive with 512B sectors to avoid BIOS/NVMe boot challenges, rest of the system is on the NVMe with 4kB sectors).
2. Firstboot stuff fails
3. After clicking "finish" for firstboot, find out that no VM will start successfully (which explains firstboot failures I guess)
4. Look to the VM logs, and find this from there:

...
[ 0.887548] blkfront: xvda: flush diskcache: enabled; persistent grants: enabled; indirect descriptors: enabled;
[ 0.902355] blkfront: xvdb: flush diskcache: enabled; persistent grants: enabled; indirect descriptors: enabled;
[ 0.924386] blkfront: xvdc: flush diskcache: enabled; persistent grants: enabled; indirect descriptors: enabled;
[ 0.940325] blkfront: xvdd: flush diskcache: enabled; persistent grants: enabled; indirect descriptors: enabled;
Waiting for /dev/xvda* devices...
Qubes: Doing R/W setup for TemplateVM...
[ 1.049451] random: sfdisk: uninitialized urandom read (4 bytes read)
[ 1.052481] xvdc: xvdc1
[ 1.060250] random: mkswap: uninitialized urandom read (16 bytes read)
Setting up swapspace version 1, size = 8 GiB (8589930496 bytes)
no label, UUID=...
Qubes: done.
mount: wrong fs type, bad option, bad superblock on /dev/xvda,
 missing codepage or helper program, or other error

 In some cases useful info is found in syslog - try
 dmesg | tail or so.
Waiting for /dev/xvdd device...
mount: /dev/xvdd is write-protected, mounting read-only
[ 1.099814] EXT4-fs (xvdd): mounting ext3 file system using the ext4 subsystem
[ 1.106796] EXT4-fs (xvdd): mounted filesystem with ordered data mode. Opts: (null)
mount: /sysroot not mounted or bad option

 In some cases useful info is found in syslog - try
 dmesg | tail or so.
[ 1.119049] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x1e335a008d5, max_idle_ns: 440795216613 ns
mount: /sysroot not mounted or bad option

 In some cases useful info is found in syslog - try
 dmesg | tail or so.
switch_root: failed to mount moving /sysroot to /: Invalid argument
switch_root: failed. Sorry.
[ 1.217841] Kernel panic - not syncing: Attempted to kill init! exitcode=0x00000100
...

**Expected behavior**

VMs would start. Firstboot stuff would work. Drives with 4kB sector size would work.

**Additional context**

I've tracked this down to the handling of the partition table. With 512B sectors the location of the GPT differs from that of with 4kB sectors and therefore VMs fail to find the correct partition table from xvda. Obviously also the partition start/end values will be off by the factor of 8 because the templates are built(?) with an assumption of 512B sector size.

I'm not sure if there are other assumptions based on 512B sectors with the other /dev/xvd* drives.

**Solutions you've tried**

I cloned a template and I tried to manually fix the partition table of the clone (in dom0 through /dev/qubes_dom0/...). There's was plenty of space before the first partition, however, at the end the drive is so tight on space that the GPT secondary table won't fit so the xvda3 partition's tail was truncated slightly and I didn't try to resize its filesystem first (this probably causes some problems, potentially corruption?). With such a fixed partition table, I could start VMs (but there are then some other problems/oddities that might be due to incomplete firstboot or non-fixed fedora template, I only fixed the debian one which I mainly use normally). I could possibly enlarge the relevant LV slightly to avoid the truncate problem at the tail of xvda3 but I've not tried that yet.

I tried to look if I could somehow force pv/vg/lv chain to fake the logical sector size but couldn't find anything from the manpages.

Libvirt *might* be able to fake the
logical_block_size
 but I've not yet tried that.

**Relevant documentation you've consulted**

During install, I used the custom install steps to create manual partitioning (but I think it is irrelevant).

**Related, non-duplicate issues**

None I could find, some other issues included failure to mount root successfully but the causes are different.

**Decided solution**

Add a partition table conversion to initramfs. Specifically, write a tool that would check if partition table matches current block size. If it matches, do nothing. If not, convert it to the right block size format before mounting anything. And destroy the wrong partition table (if isn't directly overridden by the converted one) to prevent confusion which one is the current one.

References:
 URL
 URL

A lot of useful info: URL
There is also a script to parse 512-byte GPT on 4k disk (and map it using loop devices).
Using this, one workaround would be to adjust `init.sh` to rewrite GPT if sector mismatch is detected (in either direction). This require the partitions to be 4k aligned before, but it should be doable.
But this is far from a complete solution, given non-template-based Linux use cases.

There's not much to worry about 4k alignment, it is already there in the template: what I gathered, the partition table tools nowadays will enforce at least 4k alignment and they will warn if that would be violated (some might do even larger alignment). This is why I managed to rewrite the template's partition table in the first place so easily (except the truncate issue).

I don't think forcing 512 sector size itself would come with a large penalty as in practice the filesystems inside will use something larger than 512 (depending on how all relevant block stuff handles the larger continuous units of course but I'd guess that would not cause performance problems). So it would be mostly relevant for booting up correctly. What I'd rather avoid though, is forcing my drive's firmware to use 512 sector size as it would explore less tested corners of the firmware and possibly have significant performance impact too (I know my NVMe drive could do 512 but I don't know if all 4k drives are able so this needs to be handled anyway).

Btw, the USB HDDs might expose 4kB when not behind the SATA-to-USB converter, perhaps you have one of them which you can disassemble to get such a device?

Like I said, libvirt supposedly has a way to configure
logical_block_size
 but I don't know if that is able to fake it for real:
 URL
...or is that only for KVM?

I'll probably try to use the file backend (that's what was used in R3.2 right?) for the main system for now (the NVMe drive should clone fast anyway :-) so the biggest downside I know of is a non-issue). Can the installer do that automatically if I simply reinstall (that is, how it chooses which type of storage pool to use by default) or do I have to manually setup everything afterwards skipping the firstboot stuff to avoid it failing? I can then look into the 4k stuff while other stuff keeps working fine with 512. That would also allow me to easily test cross 512 and 4k copying but that looks rather scary to begin with, so far about nothing seems portable from one sector size to another from what I've read.

If the partition table would be removed from xvda, the grub might have a similar 4k vs 512 issue anyway so that might not solve anything (sector was mentioned somewhere when I tried to look into what kind of information format it uses which sound bad) but this needs a deeper investigation.

Yes, I think it's KVM only.

If you choose btrfs during installation, Qubes will use that instead of LVM.

Could not the faking be done other way around? I'd feel by intuition that in block code log4096 -> phy512 is far simpler than log512 -> phy4096.

Or is there some particular reason why 512 is still needed for the VM disk format that is almost internal to Qubes. VMs will obviously see the end result but they should have little reason to change how the sector size is defined by the "internal" format. Or is there some other OS that only works with 512?

That would leave just a few things to address:
- block support for log4096 -> phy512 (or any 2^n to be more generic).
- Template building code to create 4k GPT (sfdisk doesn't seem to support forcing sector size unlike some other partition tools but losetup could be used to fake it).
- Fix the 512 assumptions (such as the one with swap partition sizing), hopefully not that many
- One-time migration (at new release?) which would be at most as complex as a GPT rewrite workaround needed for supporting both sizes, probably somewhat less.

I'm not sure about disks emulated by QEMU. And then windows PV drivers. Recently I've seen some patches flying around fixing 512 sector size assumption somewhere there, so there still may be more issues like this.
Given various elements involved, I think 512 is simply safer in terms of compatibility.

Can I throw in another alignment data point to consider: the LVM chunk_size, which can range from 64KB to 1MB.

Policy-wise, Qubes may want to consider ensuring that any physical partitions (or partitions inside lvm LVs), that are created by qubes tools, are 1MB aligned, primarily for performance reasons. Probably not as critical as the baseline fixes to ensure 4K logical sector drives work, but since that requires changes, consider enforcing a much more strict alignment going forward (see the volatile volume issue #5151).

Brendan

If anyone needs 4Kn templates right now, can use my patch from URL

I have also found that ``partprobe`` will fail to spawn the partitions on loop devices created with the custom sector size (losetup: -b / --sector-size) not corresponding to the sector size of the backing disk on Linux `< 4.18-rc4``.

This will become a bigger problem with R4.2, where cryptsetup >= 2.4.0 (Fedora >= 35) will automatically create dm-crypt with 4K logical sectors even on 512e drives.

Ideas:

- Invoke `cryptsetup luksFormat` with an explicit `--sector-size=512` argument for the LVM Thin installation layout (fix for 512e drives)

- Attach thin volumes via a loop devices (fix for 4Kn and 512e drives)

I wonder if Qubes pools should specify the sector size of their underlying storage technology, and whether importing volumes should involve a conversion step?

B

Conversion during import would mean parsing VM data in dom0 😬

Ok someone should definitely write a DisposableVM-powered converter for common volume contents.

But automatic conversion won't be possible in all cases (like standalone HVMs where a volume could contain anything, e.g. bs dependent filesystems that might not be straightforward to upgrade), so even with a very good converter there's still a need for

- per-volume metadata recording the appropriate bs for its current content
- a mechanism to present the volume to the VM with that bs, even if the storage pool's ideal bs is different, e.g. after restoring from a backup

I'm kinda curious too about how writes really work for kernel -> 512e drive communication.

Pure speculation: Since both the kernel and the drive know that the drive's physical block size is 4K, maybe the kernel just always writes batches of 8 * 512B logical blocks - and when the drive sees logical blocks coming in fast enough, one immediately following another, it figures out that read-modify-write can be avoided? Or there could be some explicit way for the kernel to signal to the drive that it's aware of 512e and that it guarantees to send 4K blocks merely encoded as batches of 512B blocks.

Huh. Thanks! Wonder if that's better than a loop device.

I can think of at least two solutions:

1. Place the partition table for a 512e device between the protective MBR and the 4Kn GPT. There are 7 512-byte sectors in this space, which allows for up to 6 partitions. This is enough to fit all three partitions used by Qubes OS, plus one extra covering the 4Kn partition table. The only problem with this approach is that if the block size is 4K, the protective MBR will appear to extend past the end of the device. I suspect this is harmless.
2. Use a partition table that is not part of the image, but instead is overlayed on it at runtime. This can be done by using `dm-linear`.

@DemiMarie I don't get (1). Would there be some script in the VM's initrd to rewrite the partition table ("activating" the stashed away 512B version) depending on xvda's current logical block size?

Dynamically switching back and forth between 512B and 4K partitioning seems like it could make resizing the volume (`resize-rootfs-if-needed.sh` and `resize-rootfs`) a little scary...

Generally, I'd try to avoid any kind of conversion at startup and go for emulation when necessary. That means:
 - recording expected block size by the volume
 - emulating it, if the underlying pool has different block size

And then, either build templates with two flavors, or convert at the install time (as part of `qvm-template-postprocess`), if reasonably easy.

Can we get away without emulating 4k bs on 512 bs devices?

Once there's a way to attach a volume as 4K, why even bother building (or converting to) 512B templates.

Forcing `--sector-size=4096` for `luksFormat` in the installer (or `reencrypt` in the upgrade script) even on drives reporting 512B physical sectors would have the same effect.

I'd guess almost all of those drives (that make sense to install a Qubes storage pool on) actually have 4K physical sectors anyway, but it's misreported by shoddy firmware or an adapter.

Ah wait, I haven't tested if `reencrypt --sector-size=4096` would work to bump up an *existing* LVM Thin pool. I'll try it today or tomorrow.

Thanks :roll_eyes: Samsung! I just checked `/sys/block/DEVICE/queue/*_block_size` for an older consumer 850 Pro and indeed it's 512 logical / 512 physical where my daily driver, a datacentery PM883 says 512/4096. Both SATA.

But a consumer NVMe by Western Digital also said 512/4096.

Come to think of it, was that ever something to be relied on? Like, even in a pure 512B world of ancient spinning rust - partially written or otherwise corrupted blocks are legal (and not unusual) in case of a power outage, no? And block device users like filesystems must cope with this through journals or other suitable data structures.

Almost all storage primary devices produced today are (at least internally) logically 4096 and present externally as either 512e (emulated logical 512) or 4Kn (native).

Physically it's even weirder since now both flash *and spinning platters* on primary storage can have larger physical minimal write sizes under some/all circumstances (and much larger physical erase/zeroing sizes for flash, such as 1MB or 4MB). Also the SMR disks are super-super-weird.

But we mostly have to worry about whether the pool is on a 512n, 512e or 4096 device.

I think the safest thing to do is force everything (partitions, crypto overlays, pools, templates
Volumes, filesystems) to aligned, 4K block sizes up and down the stack.

For better erase-block-faster-after-discard support, keeping partitions/overlays/volumes 1MB or 4MB aligned might be even better.

Possibly keep around a 512 pool (or allow creation of an ephemeral 512 pool before conversion) for restoring/importing 512-based VMs but with the knowledge that those will run non-optimally until converted. And direct-io would be disabled for such a pool on a 4Kn device.

I think the future is 4Kn. Near future anyway.

B

No, both partition tables would be present at all times. The logical block size determines which one the kernel will parse.

I recommend checking that at least Linux and systemd-gpt-generator handle it properly. We will of course need to implement our own code for _writing_ these hybrid partition tables, but existing tools should have no problems _reading_ them, at least if I understood the GPT specification correctly.

Some more information:

- ext4 blindly assumes that 4K overwrites are atomic, no matter what the logical sector size is.
- XFS is aware of the logical vs physical sector size distinction. It will try to use physical-sector writes where possible, but only assumes atomicity of logical-sector writes.
- I am not sure about BTRFS, ZFS, LVM, or dm-thin. With BTRFS and ZFS at a minimum one should get an error instead of wrong data.

There was a paper (I will try to find a link later) that found that on some NVMe devices, 512B writes were atomic while 4096B writes were not.

Overall, I think these kind of problems are a good reason to switch to BTRFS-by-default, at least if BTRFS actually solves them. Since BTRFS is copy-on-write, it can (at least in theory) provide atomicity guarantees for arbitrary-sized writes, and I hope that it provides them for 4K writes at a minimum.

Moving an existing LVM Thin pool from a logical 512B block device to a logical 4K block device appears to work fine and *does* bump up the logical block size - but uh oh (#5685):

Here's a written down concrete proposal of the sketched out transition to 4K storage, so my head doesn't explode.

It would fix 4Kn drive incompatiblity (this issue), prevent the upcoming 512e drive incompatibilty, and accomodate the corner case of unencrypted (or otherwise unable to be reencrypted to 4K) lvm_thin pools on 512B drives.

Let me know what I've missed.

## 1. Add `Volume.logical_block_size` property

- default value 512
- distinguish between default value 512 and explicitly assigned value 512
- volume cloning preserves source value (default vs. assigned still distinguished)
- expose via Admin API
 - `qvm-volume config`
- `qvm-template(-postprocess)` support
 - assign value for 'root' from template package metadata (512 if missing)
- backup/restore support
 - preserve value (still distinguished) for 'root' and for 'private'
 - if feasible, preserve assigned value for 'volatile' object (see "rare case" in section 4)
 - Wyng?
- forum: notify existing users of custom built 4K templates that they should assign 4096 in preparation
 - also after they restore already backed up (before property implementation) custom built 4K templates in the future

## 2. Storage driver support for `Volume.logical_block_size`

- file(-reflink): `losetup --sector-size=...` in `/etc/xen/scripts/block`
 - old `losetup` triggers a race in old kernels with that argument
- lvm_thin: dm-ebs (can emulate in either direction)
- `Volume.ephemeral_volatile`
 - call `cryptsetup --type=plain` with `--sector-size=4096` (fc32 manpage is outdated, it works)
 - optional optimization for 512 as 4K: skip dm-ebs layer, 4K dm-crypt is enough
 - 'volatile' size must be a multiple of 4K (neither dm-ebs nor dm-crypt allow unaligned size)
 - lvm_thin: looks like LVM Thin already aligns size to 4 MiB
 - file-(reflink): loop device always aligns size to 512

## 3. Optimize Qubes OS installation

- hardcode `cryptsetup luksFormat --sector-size=4096` in installer
 - raw device size must be a multiple of 4K or `cryptsetup` will fail
 - `cryptsetup --device-size=...` appears to be broken for this purpose
 - report bug
 - work around somehow or patch
- `qubes-dist-upgrade`: optional final `cryptsetup reencrypt --sector-size=4096` phase where feasible
 - not if XFS is dom0 fs
 - tricky if raw device size isn't a multiple of 4K
 - steal bytes from the old LUKS2 header?
 - offline reencryption instructions for old LUKS1 users coming from R4.0?

## 4. Move to 4K volumes

- assign 4096 to all default valued volume objects (existing or newly created) named 'kernel'
- assign 4096 to all default valued volume objects (existing or newly created) named 'volatile'
 - dependency on new enough kernel-(latest-)qubes-vm / qubes-kernel-vm-support
 - rare case: standalone HVMs etc. doing custom partitioning of 'volatile' on startup
 - it's just 'volatile' - might be okay to break, and require user to assign 512 or fix their custom script?
- assign 4096 to all newly created volume objects (cloning/restoring might then assign something else)
- build 4K Templates
- everything else: handle easy cases, provide user guidance and converters
 - existing 'private'
 - could be custom incompatible filesystem (XFS) etc.
 - VM init script advertising `volume-private-supports-logical-block-size=4096` feature if standard filesystem detected?
 - mass assignment command for people who know they're only using normal 'private' volumes
 - existing 'root' in TemplateVMs and StandaloneVMs
 - converter scripts

That is partly covered here: URL

Which requires manual alignment and lead to open URL for misalignment issues and needs for manual calculation since sfdisk and other tools were not properly dealing with partition table and partition alignments.

Heh, as an aside, I'm fairly certain this cross-manufacturer usb-SATA bridge behavior (stealing the last sector) exists in some chipsets to support saving state (likely not used by most variations of the bridges, even, but e.g. encryption/password) without having to include eeprom or NOR flash on the chipset. Possibly to ensure portability of that state with the drive (within the chipset anyway).

The amount of chaos it was to cause, however, was likely not forecast by the original engineers...

If not done already, it would be wise to ensure the remnant non-alignment friendly tails at the end of the storage devices are always excluded from usage via partitioning excluding them.

B

Thanks! Latest updates to the proposal:

- workaround for `cryptsetup` is to ensure good partitioning
- clarified default value mechanism for the property, it was too vague
- legacy 'file' driver doesn't use `/etc/xen/scripts/block`, but linux-kernel driver does

Here is openQA run with 4kn emulated disk: URL
fdisk there reports:

fdisk -l /dev/nvme0n1
fdisk -l /dev/nvme0n1
Disk /dev/nvme0n1: 80 GiB, 85899345920 bytes, 20971520 sectors
Disk model: QEMU NVMe Ctrl
Units: sectors of 1 * 4096 = 4096 bytes
Sector size (logical/physical): 4096 bytes / 4096 bytes
I/O size (minimum/optimal): 4096 bytes / 4096 bytes

And every VM fails to start, as expected.

Given the failure mode on 4.2 is worse than on 4.1, I think we should have it in 4.2. The plan outlined by @rustybird in URL looks good. @rustybird are you up for implementing this?

@marmarek: what about write tearing? 4K sector writes on a 512e driver are not guaranteed to be atomic, and IIRC are _not_ atomic on some low-end SSDs in the event of power failure. XFS takes precautions against this.

Maybe we should get the R4.2 regression affecting 512e disks out of the way first, by hacking the installer to use `cryptsetup luksFormat --sector-size=512` whenever the LVM Thin layout has been selected and the destination disk has a 512-byte logical block size.

But yes I'm also still interested in starting on the proposal's phase 1 and 2 at least, to fix the existing lvm_thin (and zfs driver too?) incompatibility with 4Kn drives. Not sure how long it will take though.

Phase 3 and 4 is where things would get spicy with the whole atomicity question. If a Qubes storage volume is exposed to the VM as a 4K block device even if the disk hardware might not provide atomic 4K writes, will this cause the filesystem on the volume to falsely *rely on* 4K writes being atomic *when it otherwise wouldn't have* - either for its own purposes in attempting to preserve its data structures' integrity, or as some sort of ineffective guarantee to an application writing data into a file? Tbd... There's an interesting writeup: URL

I tried in URL
But it didn't worked: URL

[2023-08-14 05:27:22] Waiting for /dev/xvda* devices...

[2023-08-14 05:27:22] Qubes: Doing R/W setup for TemplateVM...

[2023-08-14 05:27:23] [ 2.836815] xvdc: xvdc1 xvdc3

[2023-08-14 05:27:23] Setting up swapspace version 1, size = 1073737728 bytes

[2023-08-14 05:27:24] [ 3.052640] random: crng init done

[2023-08-14 05:27:24] UUID=50685a9b-25ff-4cb6-b107-7170602a08e6

[2023-08-14 05:27:24] Qubes: done.

[2023-08-14 05:27:24] mount: mounting /dev/mapper/dmroot on /sysroot failed: Invalid argument

[2023-08-14 05:27:24] Waiting for /dev/xvdd device...

[2023-08-14 05:27:24] [ 3.086464] /dev/xvdd: Can't open blockdev

[2023-08-14 05:27:24] [ 3.086937] EXT4-fs (xvdd): mounting ext3 file system using the ext4 subsystem

[2023-08-14 05:27:24] [ 3.089277] EXT4-fs (xvdd): mounted filesystem with ordered data mode. Quota mode: none.

[2023-08-14 05:27:24] mount: mounting none on /sysroot/lib/modules failed: No such file or directory

[2023-08-14 05:27:24] [ 3.172895] EXT4-fs (xvdd): unmounting filesystem.

[2023-08-14 05:27:24] mount: can't read '/proc/mounts': No such file or directory

[2023-08-14 05:27:24] BusyBox v1.36.0 (2023-01-10 00:00:00 UTC) multi-call binary.

[2023-08-14 05:27:24]
[2023-08-14 05:27:24] Usage: switch_root [-c CONSOLE_DEV] NEW_ROOT NEW_INIT [ARGS]

[2023-08-14 05:27:24]
[2023-08-14 05:27:24] Free initramfs and switch to another root fs:

[2023-08-14 05:27:24] chroot to NEW_ROOT, delete all in /, move NEW_ROOT to /,

[2023-08-14 05:27:24] execute NEW_INIT. PID must be 1. NEW_ROOT must be a mountpoint.

[2023-08-14 05:27:24]
[2023-08-14 05:27:24] -c DEV Reopen stdio to DEV after switch

[2023-08-14 05:27:24] [ 3.193656] Kernel panic - not syncing: Attempted to kill init! exitcode=0x00000100

I think I did set it correctly:

root# cryptsetup luksDump /dev/nvme0n1p3; echo efDdj-$?-
LUKS header information
Version: 2
Epoch: 3
Metadata area: 16384 [bytes]
Keyslots area: 16744448 [bytes]
UUID: c51bf27f-c7e1-443b-917d-b95db5145101
Label: (no label)
Subsystem: (no subsystem)
Flags: (no flags)

Data segments:
 0: crypt
 offset: 16777216 [bytes]
 length: (whole device)
 cipher: aes-xts-plain64
 sector: 512 [bytes]
...

Any ideas?

@marmarek:

The test had `HDDMODEL` configured with `physical_block_size=4096,logical_block_size=4096`, so it's emulating a 4Kn drive. To test 512e (in the sense that's causing the R4.2 regression due to new cryptsetup) it should be `physical_block_size=4096,logical_block_size=512`

(Hardcoding `--sector-size=512` doesn't help with 4Kn drives, because dm-crypt cannot shrink the logical block size, it can only avoid enlarging it.)

Is it only partition table issue? What about the filesystem?
But also, the remark about dynamic resize is a valid one. Only one partition table will be updated, so if you migrate to a disk with a different logical block size, you'll get truncated partition.
But if that's _just_ about partition table, maybe we can do the conversion in initramfs before mounting anything? Assuming everything is 4k-aligned, it should be technically possible, right?

Yup, it’s just about partition table, and we can avoid the dynamic resize problem by changing the partition table with our own tools that understand the different layout.

@rustybird what do you think about adjusting partition table in initramfs?

I like the simplicity of it, especially compared to my 4-phase slog of a proposal.

The adjustment script should probably bail out early if the root volume looks too nonstandard? E.g. if xvda3 is not an ext4 filesystem (or another filesystem type that's whitelisted as known to be logical-block-size agnostic).

Setting aside the question of what a good default would be for Qubes OS, using btrfs instead of LVM + ext4 actually does work around this issue on my 1 TB NVMe SSD (the model is a Sandisk Corp WD Blue SN570). It reports 4096 for both logical and physical sector size due to me following the instructions in the ArchWiki entry for Advanced Format a while back.[^1]

Anyway, I ran into the same issue that OP mentioned while installing R4.2 on my desktop using the default disk configuration scheme, but everything works OK so far on a reinstall using btrfs. I wouldn't have even thought to do this if not for this issue and this comment on a duplicate issue.

It makes me wonder whether using other filesystems that have similar properties to btrfs would work, such as ZFS. Seems like adjusting the drive to emulate a sector size of 512 again may not be a viable workaround based on others' comments here, but I haven't tested it.

[^1]: I'm aware that Advanced Format is a hard drive specific thing, but that's what the wiki entry is called.

That’s really interesting, but it actually makes sense: since BTRFS is copy-on-write, it can (at the expense of performance) make arbitrarily small writes atomic.

I currently plan to use the rust-gptman crate for this, as it is packaged in Fedora 41 and actually has things like test suites. It can be backported to older distros by bundling it.

Looks like _both_ `gptman` and `gpt` have the same bug: neither includes necessary calls to `fsync()`, and so both risk leaving the system unbootable in the event of power loss!

@marmarek: can the tool that converts from 512 to 4K and back be provided with the previous sector size (perhaps from metadata somewhere), or does it need to guess the previous sector size? The latter could be problematic if there are multiple GPTs on the disk.

You can get the current sector size (from sysfs?) and check if the current GPT matches. If not, do the conversion.

For those coming to this issue, if the device supports going back to 512 bytes then you can apply the commands in URL as a temporary solution until this is fixed (currently using it myself and don't notice any performance issues, so far it has been even faster than 4k and btrfs)

Note to self: if we ever put something on the EFI partition of the template's root volume, create filesystem there with 4k block size.

Does this apply to every Template and dom0 automatically (also ones on BTRFS) or does the user need to do something (once r4.3 drops)?

@Atrate:

The changes (thanks to @DemiMarie) generally make up-to-date template-derived VMs compatible with volume data exposed to the VM as having 4K logical sectors. This is done by rewriting the partition table inside of the "root" volume's data on early VM startup, because partition tables are typically the only kind of volume data that is not agnostic to the logical sector size.

However, the exposure of volumes to the VM as having 4K logical sectors currently only happens if it's required by the underlying storage. Automatically, it only happens for LVM on native 4K drives (drives with a 4K logical, not just physical sectors), a scenario that was previously incompatible with Qubes OS.

As a performance optimization, it would now be possible to use 4K sectors for the underlying dm-crypt storage layer in more cases:

- For LVM, it might make sense now to remove the installer patch that tries to set up 512B logical sectors for dm-crypt even if the drive reports 4K physical sectors. This would then also use 4K logical sectors for dom0.

- For Btrfs/XFS, 4K logical sectors for dm-crypt are already used if the the drive reports 4K physical sectors. However this only benefits dom0 at the moment. VMs still see 512B logical sectors, because the `qubes-block` utility always sets up loop devices with 512B logical sectors. It could be changed to use 4K logical sectors, through explicit configuration (e.g. by an environment variable) or ideally detected automatically.


Question: Liferay packages not loading from @liferay/cli generated remote app when using live development?

The problem is in this line

It should say:

javascript
{
 include: project.assetsDir.asNative
}

instead.

Liferay packages not loading from @liferay/cli generated remote app when using live development
### Issue type (mark with `x`)

- [ ] :thinking: Question
- [x] :bug: Bug report
- [ ] :gift: Feature request
- [ ] :woman_shrugging: Other

### Version (mark with `x`)

- [ ] :two: v2.x
- [x] :three: v3.x

### Description

**Desired behavior:**
Deployed/Live session remote app generated via `liferay@cli` shows correctly.

**Current behavior:**
Generated project using `liferay@cli`, choosing Portal 7.4 and Remote App with React.
When using `liferay start` the remote app is deployed but can't load external scripts (for example `react.js`). It tries to load them from the webpack server (`localhost:8081/o/frontend-js-react-web/__liferay__/exports/react.js`) where it looks like they dont exist instead of Liferay (`localhost:8080/o/frontend-js-react-web/__liferay__/exports/react.js`). Thanks to these remote app doesnt show at all.

My workaround for now is to use absolute url in `liferay.json` such as ` URL instead of provided `/o/frontend-js-react-web/__liferay__/exports/react.js`.

**Repro instructions (if applicable):**
1. Generate project using `liferay new`, targeting Portal 7.4, Remote App with React.
2. Run `liferay start`
3. Add remote widget to page
4. Remote widget does not show at all
**Other information (environment, versions etc):**
Portal 7.4 49GA
Node 16.18.0

I have created a project with the latest version of `@liferay/cli` and it works seamlessly for me:

image

See in the attached screenshot how `npm run start` starts and correctly reports that it will be fetching `react` and `react-dom` from Liferay Portal.

Also, see in Firefox (inside the screenshot) how the path to the .js files in the top import statements is correctly pointing to `/o/frontend-js-react-web/__liferay__/exports/....`, which is the path where those .js files live inside Liferay Portal.

Can you please follow the same steps to see how the built `index.js` looks like in your case?

You just need to execute `npm run start`, then open ` URL in a browser. That will show how `@liferay/cli` (and webpack) is generating that file and how the imports from `react` and `react-dom` look like.

Alternatively, you can run `npm run build` and see how the generated zip file looks like (though I'm afraid you will be affected by URL :-/ ).

I can check whenever I will have a moment. Should be today.

But I remember the problem wasn't about the js file itself. It was when I placed the remote app widget into liferay page. It tried to load from the `/o/frontend-js-react-web` from ` URL not from the liferay which resides in `localhost:8080` by default. And it didnt load from the webpack server.

Yes, but that's very strange, because if you look at the screenshot, the generated `index.js` imports `react` from `/o/frontend-js-react-web/...` without anchoring it to any host or port.

Thus, when you include that `index.js` file inside Liferay Portal and fetch it from, say, ` URL the unanchored `import` must be fetched from the page base URL (` URL not from ` URL as it is not the address you are using in the browser address bar.

In other words, `import` URLs are relative to the page (address bar URL), not to the URL from where the script was fetched. So there's no way the browser can use ` URL instead of ` URL unless:

1. It is specifically specified in the .js file
2. You are fetching a proxied Liferay Portal from ` URL

Allright, I tried newly generated project on Windows with `liferay new`.

Target platform: Liferay Portal CE 7.4
Shadow dom: No

`npm install`, then `npm run start` with the same versions reported on top.

Then I removed all rules from `styles.css` to work around the bug.

After that I dragged the remote widget to home page and it just tries to load from `localhost:8081`

This is `localhost:8081/index.js`
index.js
Some more info
image

In terminal everything is green webpack compiled successfully.

I've downloaded your version of Liferay Portal and I'm able to reproduce it too :+1: .

Which makes me think this statement is wrong:

But how has it worked until now, then? :thinking:

From MDN:

Which is ambiguous, because "site root" can be the page's site root or the .js module site root when two servers are involved.

I can make it fail with master too (in Linux). I suppose this has never worked, then :shrug:

However, I would have sworn that I saw this working before. Maybe we've been deploying projects using the _agnostic_ platform only.

Or maybe we haven't tested this feature since we stopped using import maps.

The problem with this is that it not only affects `npm run start`, but in theory, also any remote app wishing to consume `react` from Portal and being served from an external server.

Interesting - that is true they would need to provide absolute URLs as a workaround.

So, after reflecting about this a bit, I think the only possible solutions to this are:

1. Use import maps (but we can't because some browsers still don't support them and the shim produces double executions of some modules due to the way it works).
2. Specify the base URL of external imports at build time

The best solution would be number 1, because sometimes it is not possible to know where different things will be deployed at runtime and import maps lets us wire things together at deploy (or even run) time.

The thing is that, in a distributed setup, you may want to import external modules not only from portal, but also from other hosts. Imagine, for example, that you have three servers:

1. myportal.com
2. mycustomelements.com
3. mysharedlibs.com

And you want to host custom elements at number 2, and some utility libraries at number 3, then use `react` from the Portal. In this scenario, there's no way to make this work without either import maps, or hardcoding the server URLs at build time in the produced artifacts so that the browser knows where to get the libs, custom elements, and `react` from.

After some internal brainstorming, for now we think this is only a problem with `npm run start` and not a problem with real production deployments as explained in my two previous comments.

This is because 99% of the production setups will be using a unique domain name for the host (whether or not they use a CDN) since otherwise, they will need to take care of CORS, https certs, and make sure to wire things properly which, AFAIK, apart from webpack federated builds, nobody is doing on a regular basis, because there are no tools aimed at that.

If, in the future, we think that a multihost setup makes sense, we can either tell users to tweak their liferay.json files or implement some feature in @liferay/cli to make those setups easier.

----

As to the `npm run start` feature, we will probably need to ask for the URL of the local Liferay installation (in addition to the deployment directory) and either:

1. Make a specific build for live development that prepends that URL to the external imports
2. Proxy the Portal through webpack dev server so that everything comes from the same host (that would be `localhost:8081` instead of a mix of `localhost:8080` and `localhost:8081`).

Solution 2 may be better because it would also prevent any possible CORS issue that could arise because of using two different hosts in the same page.

This may be fixed once we find out how to share code between client extensions, which is being investigated at URL

Liferay/CLI project fails on scss files on Windows
### Issue type (mark with `x`)

- [ ] :thinking: Question
- [x] :bug: Bug report
- [ ] :gift: Feature request
- [ ] :woman_shrugging: Other

### Version (mark with `x`)

- [ ] :two: v2.x
- [x] :three: v3.x

### Description
Newly generated project with `liferay new project-name` choosing Portal 7.4 CE, remote app with react.
When I run `liferay build` I get an error.

You may need an appropriate loader to handle this file type, currently no loaders are configured to process this file

I skimmed through the building code and I suspect this has to do with webpack configuration and paths. I suspect it has to do with
getWebpackConfiguration and with the `getScssEntryPoints` function.

On WSL (Windows Subsystem For Linux) it runs fine.

**Desired behavior:**
Build completes.

**Current behavior:**
Build doesnt complete.

**Other information (environment, versions etc):**
Windows 10
Node 16.18.0

I've been able to reproduce this bug inside `npm run start` (should be equivalent to `npm run build`):

image

(Sorry for the screenshot, but I cannot copy text from my virtual machine :-( )


Question: Using print_rich on a text containing a newline wrapped in the [color] BBcode tag doesn't strip away the closing tag.

Docs are correct, `color=tomato` will only show in the editor log (since it's RichTextLabel and support all colors), but won't show in the OS terminal.

Using print_rich on a text containing a newline wrapped in the [color] BBcode tag doesn't strip away the closing tag.
### Tested versions

- Reproducible in Godot v4.3.stable (bce457cec)

### System information

Godot v4.3.stable (bce457cec) - Windows 10.0.22631 - Vulkan (Forward+) - dedicated NVIDIA GeForce GTX 1660 SUPER (NVIDIA; 31.0.15.2802) - AMD Ryzen 5 5600X 6-Core Processor (12 Threads)

### Issue description

When using `print_rich` to print a string that contains a [color] BBcode tag - if the string wrapped in that tag contains a newline ("\n"), the closing `[/color]` tag won't be stripped away, resulting in an output where the color works, but the "[/color]" tag remains as part of the printed string.

### Steps to reproduce

1. Create a string that has a "\n" in it.
2. Wrap it in a color tag like `[color=yellow]`.
3. Output it with print_rich.

### Minimal reproduction project (MRP)

I guess there's no need to upload a whole MRP, this can be tested with 1 line of code, try it out:

gdscript
func _ready() -> void:
 print_rich("[color=yellow]Test\ncolor tag with newline.[/color]")

image

Also reproducible in Godot 4.0.4

I am going to try to fix it

Funny fact. It doesn't strip BBcode if it is printed into Godot Console but it is stripped when it is printed into OS console
screen-1728545875

It is not only not stripped away but also isn't considered as fallback to white text which affects following text...

screen-1728546548

The os console still shows correct results:
screen-1728546564

The editor Output panel uses the BBCode implementation from RichTextLabel to display it, while terminal output uses a custom BBCode-to-ANSI conversion routine that only supports a subset of BBCode and behaves differently. See URL

I wouldn't say it's correct behavior, it's just that the BBCode-to-ANSI conversion routine *always* replaces `[/color]` with a "reset color" tag, even if `[color]` was never opened.

Likely not intended behaviour, the code inside that custom parser explicitly tries to replace `[/color]`.

if (p_string_ansi.contains("[color")) {
 p_string_ansi = p_string_ansi.replace("[color=black]", "\u001b[30m");
 ...
 p_string_ansi = p_string_ansi.replace("[/color]", "\u001b[39m"); // On this line
}

While auto-close behavior is "correct" for single color print, it isn't correct in multi color print like in my example because user will expect to see white printed in white color.

Looks like it is intended. As I understand the line you pointed resets color to default when sees closing tag so it is likely normal because if color is already default nothing changes

I just meant that it looks like it's trying to replace the closing tag, but somehow the closing tag is still showing in the output.

The code that prints to os terminal is different from code that prints to godot console. And here you are viewing the code that prints to os terminal. If you will comment the line of code you commented, your is terminal will print the same result as in godot console. `"\u001b[39m"` is an escape code for terminals

Add support for converting arbitrary BBCode colors to truecolor ANSI escape codes in `print_rich()`
### Tested versions

Present in all 4.x docs,
haven't checked 3.x.

### System information

N/A

### Issue description

The docs explicitly say only these colors are supported in `print_rich`:
Godot_v4 3-dev2_win64_PzIxbWjGiJ
even though all others also seem to be working fine.

### Steps to reproduce

`print_rich('[color=tomato]tomato color![/color]')` - Works fine

### Minimal reproduction project (MRP)

N/A

I was working on implementing arbitrary color code conversion to truecolor ANSI escape codes, but I couldn't get it to work yet.

Oh okay I wasn't aware that this was related to the terminal! The fact about bbcode to ansi conversion is only mentioned a bunch of lines below so I thought that it applied to the Godot output.


Question: Segmentation fault 11 on macOS M1 machine running pwsh -c or -file?

The repro appears to be simple enough, if we have the right profile data file to cause the seg fault. When the seg fault is seen, and is reproducible with a simple pwsh command, could you please collect the profile data file and share it? I figure it may be reproducible elsewhere too with the same profile data file.

Segmentation fault 11 on macOS M1 machine running pwsh -c or -file
### Prerequisites

- [X] Write a descriptive title.
- [X] Make sure you are able to repro it on the latest released version
- [X] Search the existing issues.
- [X] Refer to the FAQ.
- [X] Refer to Differences between Windows PowerShell 5.1 and PowerShell.

### Steps to reproduce

I'm not entirely sure this would be easily reproducible on somebody else's M1. My M1 build machine was working just fine as a Jenkins build agent until I rebooted it this evening. Then my builds started hanging on this node where it first tries to run a PowerShell script. After some debugging, I found out that trying to launch `pwsh -c ' '` causes a segmentation fault. Same with `-file`. What's odd is that I can start `pwsh` just fine:

image

### Expected behavior

console
Well, PowerShell should not crash, right?

### Actual behavior

console
I'm seeing a segmentation fault.

### Error details

console
% pwsh -nop -c 'get-date'
zsh: segmentation fault pwsh -nop -c 'get-date'

### Environment data

powershell
Name Value
---- -----
PSVersion 7.3.1
PSEdition Core
GitCommitId 7.3.1
OS Darwin 21.6.0 Darwin Kernel Version 21.6.0: Sun Nov 6 23:29:57 PST 2022; root:xnu-8020…
Platform Unix
PSCompatibleVersions {1.0, 2.0, 3.0, 4.0…}
PSRemotingProtocolVersion 2.3
SerializationVersion 1.1.0.1
WSManStackVersion 3.0

Also:

 % which pwsh
/usr/local/bin/pwsh

% ls -al /usr/local/bin/pwsh
lrwxr-xr-x 1 root wheel 38 Jan 20 21:42 /usr/local/bin/pwsh -> /usr/local/microsoft/powershell/7/pwsh

# I also have several versions of the .NET SDK and Runtime installed on this system. That shouldn't matter, right?

% dotnet --info
.NET SDK:
 Version: 7.0.100
 Commit: e12b7af219

Runtime Environment:
 OS Name: Mac OS X
 OS Version: 12.6
 OS Platform: Darwin
 RID: osx.12-arm64
 Base Path: /usr/local/share/dotnet/sdk/7.0.100/

Host:
 Version: 7.0.0
 Architecture: arm64
 Commit: d099f075e4

.NET SDKs installed:
 6.0.403 [/usr/local/share/dotnet/sdk]
 6.0.404 [/usr/local/share/dotnet/sdk]
 7.0.100 [/usr/local/share/dotnet/sdk]

.NET runtimes installed:
 Microsoft.AspNetCore.App 6.0.11 [/usr/local/share/dotnet/shared/Microsoft.AspNetCore.App]
 Microsoft.AspNetCore.App 6.0.12 [/usr/local/share/dotnet/shared/Microsoft.AspNetCore.App]
 Microsoft.AspNetCore.App 7.0.0 [/usr/local/share/dotnet/shared/Microsoft.AspNetCore.App]
 Microsoft.NETCore.App 6.0.11 [/usr/local/share/dotnet/shared/Microsoft.NETCore.App]
 Microsoft.NETCore.App 6.0.12 [/usr/local/share/dotnet/shared/Microsoft.NETCore.App]
 Microsoft.NETCore.App 7.0.0 [/usr/local/share/dotnet/shared/Microsoft.NETCore.App]

Other architectures found:
 x64 [/usr/local/share/dotnet/x64]
 registered at [/etc/dotnet/install_location_x64]

Environment variables:
 Not set

global.json file:
 Not found

Learn more:
 URL

Download .NET:
 URL

### Visuals

image

After 4 days of using 7.3.0 in my Jenkins pipeline (pwsh -c), 7.3.0 is now seg faulting on this command!

image

WTF? @SteveL-MSFT Any suggestions on a potential workaround - other than uninstalling and installing a different version? That only seems to work for a few days. This is impacting our product devops because we use PowerShell to orchestrate build & test on Mac M1.

I removed 7.3.0 and installed 7.3.2 (that must be pretty new) and now `pwsh -c` works again. Let's see how long 7.3.2 can go before it starts seg faulting. :-)

Unfortunately, not able to repro this. Was able to find two PRs in dotnet/runtime that is related to the area given the stacktrace (thanks!) URL Need to look deeper next week.

The puzzling part in the report is that the crash doesn't happen consistently -- it started to happen after using 7.3.0 for 4 days.

The stack trace suggests the crash happened when JIT start profile kicks in, which does early compilation on the methods that were used the last time `pwsh` going through the same code path -- in this case for `pwsh -c` or `pwsh -file`, non-interactive execution (_ProfileOptimization.StartProfile("StartupProfileData-NonInteractive")_).

So, it makes sense that running `pwsh` itself works, because that will be interactive execution, which will be using a different profile: _ProfileOptimization.StartProfile("StartupProfileData-Interactive")_.

@rkeithhill-keysight Can you please try the following when `pwsh -c "get-date"` results in segment fault, and see if the next run of `pwsh -c` works:
- remove the `StartupProfileData-NonInteractive` file under `/home/ /.cache/powershell`

Having this file removed will make CLR not attempt to do the early method compilation, so if this theory is correct, the next run of `pwsh -c "get-date"` should work again. If you can verify this is the case, I will open an issue on this in the .NET repo.

@daxian-dbw Yes, I'll give that a try if/when we encounter the seg fault again.

And now 7.3.2 has started to gen seg faults ~5 1/2 months after installing 7.3.2. 🤷♂

I tried @daxian-dbw suggestion to remove the `StartupProfileData-NonInteractive` file and that fixes it. Not sure how long the fix will last but my CI/CD builds are running again on this M1 mini.

@rkeithhill-keysight The root cause is in .NET, see URL Please try collecting a dump when the crash happens, and share the dump in the .NET issue -- they need a dump to further investigate the issue (see URL

I read the .NET issue **after** I had fixed it. 🤦♂ Next time this happens, I'll collect a dump and post it there.

Just ran into this after updating powershell on CentOS:

pwsh 7.3.5 (old was 7.3.4)

commands:

# update powershell
sudo yum update powershell

# Set permissions for powershell to run commands
sudo setcap cap_net_raw=eip /opt/microsoft/powershell/7/pwsh # real binary, not symlink

# rename offending config file
 mv /home/user/.cache/powershell/StartupProfileData-NonInteractive /home/user/.cache/powershell/StartupProfileData-NonInteractive.disable

FWIW it appears this issue was fixed in .NET. Here's the .NET issue and here's the PR that fixes it.

Can this be re-opend please?
I had the same issue on 7.5.0.

Disabling the file: `mv ~/.cache/powershell/StartupProfileData-NonInteractive ~/.cache/powershell/StartupProfileData-NonInteractive.disable` solved it, as suggested by @GordonVi

$psversiontable

Name Value
---- -----
PSVersion 7.5.0
PSEdition Core
GitCommitId 7.5.0
OS Ubuntu 22.04.5 LTS
Platform Unix
PSCompatibleVersions {1.0, 2.0, 3.0, 4.0…}
PSRemotingProtocolVersion 2.3
SerializationVersion 1.1.0.1
WSManStackVersion 3.0

Segmentation fault when .NET starts JIT compilation of previous recorded methods triggered by `ProfileOptimization.StartProfile`
### Description

We PowerShell team got 2 reports about `Segmentation fault` when starting PowerShell by `pwsh -command` or `pwsh -file` on Redhat 7.9 and macOS M1 machine.

- URL
- URL

The stack trace for the report on macOS M1 machine is attached below.
The stack trace suggests the crash happened when .NET starts JIT compilation of the previous recorded methods, which is triggered by the ProfileOptimization.StartProfile call in PowerShell.

We asked the user to delete the profile data file and try launch `pwsh -c` or `pwsh -file` again, and the user reported back that it began to work after removing the profile data file: URL So, it's confirmed that this is an issue with the .NET profile optimization.

image

### Reproduction Steps

The issue doesn't happen consistently. Even for the users who reported the issue, it was not consistent. They started with using PowerShell 7.3.1, which worked for a few days and then started to fail. Then they switched to PowerShell 7.3.0, which worked for a few days and started to fail again.

PowerShell 7.3.1 and PowerShell 7.3.0 are both on top of .NET 7.0.0

### Expected behavior

No segmentation fault.

### Actual behavior

Segmentation fault was reported.

### Regression?

It's a regression, given that we don't have the problem in PowerShell 7.2.x versions, which are on top of .NET 6.

### Known Workarounds

Deleting the profile data file, then `pwsh -c` and `pwsh -file` will start to work again.

### Configuration

.NET 7.0.0

Redhat 7.9 and macOS M1 machine

For Redhat 7.9: Linux 3.10.0-1160.81.1.el7.x86_64
For macOS: ARM64

### Other information

_No response_

Any updates? We got another report of the issue in PowerShell repo, and this time it's on Ubuntu 22.04

@davidwrighton @kouvel. Does Powershell use multicore JIT functionality? @daxian-dbw, is there a dump available to look at?

Yes, PowerShell calls `ProfileOptimization.SetProfileRoot/StartProfile` at the entry point.
We don't have a dump available. If you can provide steps to collect a dump in this particular case, I think @bobrozelle may be able to help with that when the issue happens again in their environment.

@mangod9

As this is a production system our focus is to not have any segfaults :-)

But i ll try to reproduce the issue on the coming weekend. Maybe i can get both, a core dump and a "corrupted" profile file.

I'll keep you up to date

Regards

@mangod9

I just reverted the permissions like described above and cache files are written now. I already got the first segfault but don't know where the dumps are, can you help me out here? I am on Ubuntu 22

Regards

I wasn't able to fully crash pwsh but some segfaults popped up:

` kernel: [187481.441049] pwsh[1565713]: segfault at 28 ip 00007f76eff09c30 sp 00007f76e8ffe8e0 error 4 in libcoreclr.so[7f76efdd7000+383000]`

Maybe i am able to test this again next weekend.

Regards

I was not able to acquire a crash or coredump, however, we have recently seen this issue occurring on a Redhat server running pwsh version 7.2.7. Removing the write permissions to the cache directory and deleting the cache files resolved the issue.

So this is also seems to be an issue for Powershell 7.2.x

Ran into this issue on AlmaLinux release 8.6, running Powershell 7.2.6. I upgraded to 7.3.6 and the issue went away for about a month, now the problem is back. This is also on a server running Nagios so pwsh is being called 100+ times per minute. I do have a copy of the corrupt StartupProfileData-NonInteractive file as I simply renamed it go get things working, if that would be of help.

Just emailed that profile data file. It does not seem to matter what is run, I'm running .ps1 files directly from bash that have a #!/usr/bin/pwsh in the first line. But even running a simple command fails:
`pwsh -command Write-Output "ok"
Segmentation fault (core dumped)`

Thanks for the data @nitroed01, I'm able to reproduce the issue. Not clear yet where the problems are, I'll investigate.

I believe I'm seeing this with Paint.NET.

Steps to repro:
1) Download and install Paint.NET v5.0.11, or use the portable version. Both available here: URL . That version of Paint.NET is using .NET 7.0.12

2) Place the two files in this ZIP into `%LOCALAPPDATA%\paint.net\Optimization`:
Startup.profiles.bad.zip
<img width="733" alt="image" src=" URL

3) Try to start Paint.NET, either by double-clicking its icon or running `paintdotnet.exe` at the command-line

Expected: It works

Actual: App does not start, and you see errors in the event viewer like so:
image

If this isn't the same issue then I can file a new issue for it, but this is causing misery for a non-trivial number of people who use my app 😢 In the short-term I can just disable the use of `ProfileOptimization` and release a hotfix for the app