Question: PCRE2 support?

Thanks for the report.

I have no intuition about what the issue is. Maybe dig into uses of `Text.fromPtr`, `Text.copy`, `Text.dropWhile8`, `Text.useAsPtr`, etc.

That's great you got use out of the library for Ormolu, at least for a little while! ❤

PCRE2 support?
Does this package support PCRE2? Reading the code I can’t see any reference to PCRE2. Do you have any knowledge of another Haskell package with pcre2 support?

Hi! :wave:

I package pcre2 for Arch Linux. Since pcre is unmaintained we are working on moving to pcre2.
@sjshuck from what I understand after reading your comment, some (but probably not all) projects in the haskell ecosystem could start relying on your package instead without huge switching costs?

@dvzrv

There is an outstanding low-level bug that caused segfaults in the Ormolu project and required the authors to switch away from the library. While it's been an ambition of mine to eventually return to hs-pcre2, life has taken me in other directions in the past few years.

This does make me want to return to it; IMHO it's a good library, it just needs some attention. I can't give a time frame right now but I did want to be up front about the state of hs-pcre2.

Is it difficult to enable pcre2 as backend for this package? sjshuck/hs-pcre2.git is interesting with its compile-time checks, but it misses `regex-base` compatibility.

@amigalemming ultimately I ended up writing my own package as needed subset of pcre2 capabilities. It is only as complete as I needed for my personal project needs and I was much worse at Haskell back then so very beginner stuff, but you can find it here hpcre2. I found the foreign function interface in Haskell a lot easier than I was concerned about. Ultimately it was not as difficult as I thought to follow the pcre2 docs and code against them as a C library interface. I'm sure if I was writing it now I would have done very differently now I'm more experienced in Haskell but I had a problem to solve and got it done.

I did try to get this added to hackage at some point but it was too time consuming to do. Now it is largely present as a reference for if others need to do the same thing.

To directly answer your question @dvzrv, in theory that could happen but the API is completely different than this library; even the smallest use sites will incur a switching cost, so if the ecosystem did that then yes, that would be huge in aggregate.

Perhaps the path of least resistance will be to create a regex-pcre2 that implements regex-base. That's not a goal of mine right now but that can change.

I've had a quick look at this and it seems that it's not so much work to get this working with pcre2. I'll likely send a pull request in the next few days.

regex-pcre2 is now on Hackage and soon hopefully on Stackage as well. It links against libpcre2.

Excellent, thanks a lot, @peteryland !
I link to your new package from the (new) README here.

Well done @peteryland what a milestone! Really appreciate your efforts and know it will make a difference for so many Haskell regex projects. Much appreciated.

`Data.Text.Array.new: size overflow`
Sorry for this report being very non-minimally reproducible, please don't feel obligated to act on this in any way.

We are using hs-pcre2 to extract some information from the Hackage Hoogle database (500MB, 70k files): relevant function, and see this script for how to execute the app (this is Nix-based and hence fully reproducible, but it should be rather easily adaptable to not rely on Nix).

At some point (it still worked in May), the app started to crash after processing a few thousand files:

extract-hackage-info: Data.Text.Array.new: size overflow
CallStack (from HasCallStack):
 error, called at src/Data/Text/Array.hs:82:20 in text-1.2.5.0-B336hruu8LkEc19BqHCzAw:Data.Text.Array

This happens both with hs-pcre2 2.1.1.1 and 2.2.1 (so text-1.2 and text-2.0), but the error message for the latter goes like this:

extract-hackage-info: Out of memory

I replaced hs-pcre2 by megaparsec in URL which makes the error go away, so we are not blocked on this and I didn't investigate further what exactly is going on with hs-pcre2. But maybe this is still interesting to you or is ringing some bell.

URL
Just a remark: It's funny that `Data.Text.Array.new` takes a signed `Int` for size, _and_ calls it "overflow" when it's negative.

The way to debug it is to build `text` with `developer` flag enabled. In order to force Cabal into doing so, pin a version of `text` which **mismatches** a boot package for GHC in use.

Also pass `--ghc-options='-fcheck-prim-bounds -fno-ignore-asserts'`.

Also you can fork `text` locally and start adding `HasCallStack` constraints to track the provenance of the error.

I'm curious if the error is still reproducible.

I had another look, and I can still reproduce this (as described in the issue description; but I think it can be minimized somewhat, at the very least getting rid of the huge `ghc-lib-parser` dependency).

Building with profiling and `+RTS -xc` for stack traces as well as enabling the `developer` flag as suggested by @Bodigrim pointed to the following issue:

Stack trace:

 *** Exception (reporting due to +RTS -xc): (THUNK_2_0), stack trace:
 Data.Text.Array.CAF
 --> evaluated by: Text.Regex.Pcre2.Internal.getErrorMessage,
 called from Text.Regex.Pcre2.Internal.check,
 called from Text.Regex.Pcre2.Internal.fix1,
 called from Text.Regex.Pcre2.Internal.matcherWithEnv,
 called from Text.Regex.Pcre2.Internal.pureUserMatcher,
 called from Text.Regex.Pcre2.TH.memoMatcher,
 called from Text.Regex.Pcre2.Internal.mapMS,
 called from Text.Regex.Pcre2.Internal._gcaptures,
 called from Text.Regex.Pcre2.TH._capturesTH,
 called from Text.Regex.Pcre2.Internal.toAlternativeOf,
 called from Text.Regex.Pcre2.TH.capturesTH,
 called from Main.extractFixitiesFromFile,
 called from Main.extractHoogleInfo,
 called from Main.main

With a few more manually inserted `putStrLn`s, it turns out that this is the offending line:
 URL
Here, `cus` is `-29`, which corresponds to `PCRE2_ERROR_BADDATA`, which is returned by `pcre2_get_error_message` here URL
because of this:

This in turn is because in `matcherWithEnv` here
 URL
we check whether `result` is positive, and throw an exception (which prints the error message) otherwise. Now the question is why `result` is non-positive. In fact, it is equal to `0`, as a result of a call to `pcre2_match`:
 URL
From the docs (emphasis mine):

This is where I am stopping for now, as I am not at all familiar with the PCRE2 C API.

---

The pattern in question is

(?m)^\\s*?(? infix[rl]?)\\s+?(? [0-9])\\s+?(? (?:[^,\\s]+\\s*?,\\s*?)*?[^,\\s]+)\\s*$

in case that is useful; I have tried to produce a minimal example of matching this pattern against the subject text, but I couldn't reproduce the error there.

---

BTW:

A less hacky way to accomplish this is to use the following in your `cabal.project`:

cabal
constraints: text source

Could it be a pcre2 bug then? Looking at URL it seems we are doing everything by the book: allocate offset vector with `pcre2_match_data_create_from_pattern`, then pass it to `pcre2_match`.

I updated the PCRE2 submodule from 10.42 to 10.44 just now. Wonder if that makes a difference.

Hmm, when I build with

cabal
constraints: text source
package text
 flags: +developer

and hs-pcre2 2.1.1 from Hackage, I get

extract-hackage-info: Data.Text.Array.new: size overflow
CallStack (from HasCallStack):
 error, called at src/Data/Text/Array.hs:74:19 in text-2.0.2-f5aebd26e19e5a1c2907d840502e1f413aa9a5057431546d415964b124f1ffd2:Data.Text.Array

If I instead build from recent `master` commits (tried 2d01498056569905abc643fdffa697efbbd6c08c and 8ac889a7e0e26564f663f1edd2bffce9bbcb6f65, so with PCRE2 10.44 and 10.42), I instead get

malloc(): unsorted double linked list corrupted
fish: Job 1, 'cabal run extract-hackage-info …' terminated by signal SIGABRT (Abort)

or sometimes also

malloc(): mismatching next->prev_size (unsorted)

Maybe the investigation I posted above is a red herring, and there is some nasty undefined behavior going on in pcre2 :thinking:

URL suggests that both `malloc()` failures mean memory corruption.

Debugging with a custom `malloc()` and `free()`, I am definitely seeing that `free()` is being called on unallocated addresses.

It happens
* a variable number of times for a test of #39 (often but not always segfaulting), and
* exactly twice for the sub callout test

Okay, after analyzing lifecycles of `Ptr`s I've narrowed this down a lot.

It's a use-after-`free()` of `pcre2_match_data` in this streaming function.

Running a test of #39 (in the above branch) produces this output. This is what I get 99% of the time. Note the last two lines.

Failures:

 src/hs/Text/Regex/Pcre2/Internal.hs:969:38:
 1) basic matching produces well-formed UTF-8 matches ( URL
 uncaught exception: ErrorCall
 ****** Using deallocated 0x00007836a8000f70 for pcre2_get_ovector_pointer ******
 malloc() for pcre2_match_data
 using for pcre2_match
 using for streamYield
 using for pcre2_get_ovector_pointer
 using for pcre2_get_ovector_pointer
 ...
 using for pcre2_get_ovector_pointer
 using for pcre2_get_ovector_pointer
 using for pcre2_match
 using for streamYield
 free() for pcre2_match_data
 using for pcre2_get_ovector_pointer

I'm expecting that `withForeignPtr` will keep alive the `matchData :: ForeignPtr Pcre2_match_data`, and thus postpone the finalizer (`pcre2_match_data_free`), until the `ForeignPtr` is unreachable. Well, I think that was a reasonable expectation, except the `ForeignPtr` is unreachable a lot sooner than the underlying `Ptr` is unreachable. See this warning.


Question: No way to configure ALPN settings?

It's been a while, I'll close here, assuming there's no interest anymore. Feel free to reopen.

No way to configure ALPN settings
Hi, while trying to use `rustls-acme` with `tonic`, we ran into problems that I think are related to not having the ability to configure `rustls` ALPN settings. (Full context is here: URL

Currently, the `rustls-acme` API doesn't allow setting `rustls` `ServerConfig` at all.

Would it be preferable to expose the entire `ServerConfig` as part of the `AcmeConfig` builder API, or just the ALPN settings?

Thanks for the opening the issue. Note that `AcmeConfig` is used for both the low level and high level API in `rustls-acme`. That's why it only offers configuration of acme related things. With the low level API you should be able to do what you want already (have a look at the `low_level*.rs` examples).

But I do think it would be nice to offer at least basic alpn control via the simple API (on `Incoming`).
1. The most we can do here is allow the user to provide a `rustls_config` and simply overwrite the resolver. That would also cover other rustls config changes the user wants to make.
2. Or we could let the user specify the alpn parameters at construction (`AcmeState::incoming_with_alpn(self, cb: Fn(rustls::ClientHello) -> ...)`)

Maybe both. I'll see what I can come up with today. If you try the low level api and run into problems or have opinions/suggestions regarding the above, let me know.

I added an `alpn_protocols` argument to all functions that construct `Incoming`, which makes this available in the high-level api.
`rustls-acme = { git = " URL }`

In the long run I'm considering merging the low- and high-level APIs, by making the stream emit something similar to rustls::server::Accepted, which the user can turn into a stream using their own config. That would side-step the whole issue. But it is good to get this out the door first, so there's something to compare the new API to during development.

Please let me know if this works for you and if you have any usability concerns, so I can tag a release asap.

Amazing, thanks so much! We tested with a fork where we just hardcoded in ALPN settings and got things working, we'll try the patch now and let you know if it works.

We've tested and this works for us! If you'd be interested in #38, we could also submit a PR for that so you'd only have to do one release, but we'd also be happy to do that later if you'd prefer more deliberation. Whatever works.

Add an optional `tonic` feature with tonic compatibility helpers?
We're using `rustls-acme` with `tonic`, and so far we wrote a some boilerplatey code to hook up `rustls-acme` to Tonic's batteries-included server implementation.

If we extracted that code and submitted a PR, would you be interested in upstreaming it under a feature flag?

I'm imagining implementing an extension trait, say `AcmeTonicExt`, so that instead of writing

rust
Server::builder()
 .add_service(/* ... */)
 .add_service(/* ... */)
 .serve(addr)

for an http server, it would be possible to write

rust
Server::builder()
 .add_service(/* ... */)
 .add_service(/* ... */)
 .serve_with_acme_tls(addr, acme_config)

for an auto-https server. The `serve_with_acme_tls` method would be provided by the extension trait, and handle gluing the `rustls-acme` methods to the `tonic` `serve_with_incoming` method.

I'm definitely open to this. I'm not familiar with tonic, but if you send a PR or a more detailed sketch I'll happily review.


Question: Vscode does not persist workspace state on closing?

Hey @joaomoreno, this issue might need further attention.

@degtiaan, you can help us out by closing this issue if the problem no longer exists, or adding more information.

Vscode does not persist workspace state on closing
Related issue - #160077

System Configuration
- macOS Sonoma - Version 14.5 (23F79)
- Apple M3 Chip

Vscode
- Version: 1.91.0 (Universal)
- Date: 2024-07-01T18:54:18.541Z (4 days ago)
- Electron: 29.4.0
- ElectronBuildId: 9728852
- Chromium: 122.0.6261.156
- Node.js: 20.9.0
- V8: 12.2.281.27-electron.0
- OS: Darwin arm64 23.5.0

Issue - when closing VSCode via "cmd + w" / red-close button via GUI(this way of closing is the regular way) the previously opened folders/workspaces are not persisted when re-opening VSCode

Instead after reopen every time the "Welcome page" is opened and all previous file changes / opened folders are lost and need to open them again.

If instead the VSCode window is closed via "cmd + q" / right-click -> quit via GUI then the workspaces / opened files are persisted on the next open, so the issue is only for "cmd + w" / close button quit.

Expected: -
Previously opened workspaces / files should persist on close and reopen when reopening VSCode.

Actual: -
Previously opened workspaces / files are not persisted when closing VSCode, instead "Welcome page" opens everytime.

also having this issue! M3 Chip with Sonoma 14.5, very very annoying.

I have the same issue staring from Vscode release 1.86.2 and for all releases after that.
Unfortunately my issue #205440 has been closed without fix.

BUT ISSUE STILL EXISTS.

The described behavior is how it is expected to work. If you disagree, please explain what is expected and what is not in more detail. See also our issue reporting guidelines.

Happy Coding!

Workspace state is not preserved
Type: Bug

On Linux (Ubuntu 18.04) after manual update of portable version of VsCode from version 1.85.2 to 1.86.2 (just extract VSCode-linux-x64 folder from archive and copy "/data" subfolder from the previous version) start the new version ... but state of current workspace is not restored (folder trust warning appeared, all oppened files disapeared, c++ compiler kit cleared). This issue is repeated every time when the VsCode restarted.

ERROR RECOVERY:
Returned back to the previous version of Vscode 1.85.2

VS Code version: Code 1.86.2 (903b1e9d8990623e3d7da1df3d33db3e42d80eda, 2024-02-13T19:41:37.860Z)
OS version: Linux x64 5.4.0-150-generic
Modes:

 System Info

|Item|Value|
|---|---|
|CPUs|Intel(R) Core(TM) i7-8665U CPU @ 1.90GHz (8 x 3027)|
|GPU Status|2d_canvas: enabled canvas_oop_rasterization: disabled_off direct_rendering_display_compositor: disabled_off_ok gpu_compositing: enabled multiple_raster_threads: enabled_on opengl: enabled_on rasterization: enabled raw_draw: disabled_off_ok skia_graphite: disabled_off video_decode: enabled video_encode: disabled_software vulkan: disabled_off webgl: enabled webgl2: enabled webgpu: disabled_off|
|Load (avg)|0, 1, 1|
|Memory (System)|31.01GB (22.36GB free)|
|Process Argv||
|Screen Reader|no|
|VM|0%|
|DESKTOP_SESSION|ubuntu|
|XDG_CURRENT_DESKTOP|Unity|
|XDG_SESSION_DESKTOP|ubuntu|
|XDG_SESSION_TYPE|x11|
 Extensions (46)

Extension|Author (truncated)|Version
---|---|---
cppincludeguard|aki|1.7.1
Bookmarks|ale|13.4.2
vagrant|bbe|0.5.0
cmake-format|che|0.6.11
doxdocgen|csc|1.4.0
dlt-syntax|Dem|0.0.1
githistory|don|0.6.20
gitlens|eam|14.8.2
cmake-test-adapter|fre|0.17.2
pythonsnippets|frh|1.0.2
gitlab-workflow|Git|4.2.0
go|gol|0.41.0
vscode-test-explorer|hbe|2.21.1
c-cpp-flylint|jbe|1.14.0
better-cpp-syntax|jef|1.17.2
cmake-language-support-vscode|jos|0.0.9
vscode-catch2-test-adapter|mat|4.8.3
dlt-logs|mbe|1.81.0
thrift-syntax-support|mrk|0.0.12
vscode-docker|ms-|1.29.0
vscode-dotnet-runtime|ms-|2.0.1
debugpy|ms-|2024.0.0
isort|ms-|2023.10.1
python|ms-|2024.0.1
vscode-pylance|ms-|2024.2.2
jupyter|ms-|2024.1.1
jupyter-keymap|ms-|1.1.2
jupyter-renderers|ms-|1.0.17
vscode-jupyter-cell-tags|ms-|0.1.8
vscode-jupyter-slideshow|ms-|0.1.5
remote-containers|ms-|0.338.1
remote-ssh|ms-|0.108.0
remote-ssh-edit|ms-|0.86.0
remote-wsl|ms-|0.86.0
vscode-remote-extensionpack|ms-|0.25.0
cmake-tools|ms-|1.17.15
cpptools|ms-|1.19.3
hexeditor|ms-|1.9.13
remote-explorer|ms-|0.4.1
remote-server|ms-|1.5.0
test-adapter-converter|ms-|0.1.9
autodocstring|njp|0.6.1
pokowaka-iwyu|pok|1.0.3
code-spell-checker|str|3.0.1
umlet|the|15.1.0
cmake|twx|0.0.17

I can't repro this. Not only did the scenario work fine, it also showed me the release notes:

Image

Can you provide more details?

Hi
Just recheck with new Vscode 1.87.0 ... the issue is reproduced

For current project I have to use custom linux toolchain so, use the next bash script for configure work env.

`
#!/bin/bash

# Linux x86 custom toolchain
source /home/username/MyProjects/some-workspace/custom_toolchain_folder/environment-setup-x86_64-linux

cd /home/username/MyProjects/some-workspace/
/home/username/MyProjects/Utils/VSCode-linux-x64/code my_workspace_name &
`

Sorry, I don't understand you note
 "Not only did the scenario work fine, it also showed me the release notes"

this seems is the error I'm talking about ...
after update ... every time when I start vscode is appeared
- folder trust warning ;
image

- welcome pages for updated extensions (e.g. GitLens, Bookmarks);

- next warnings
image

- all previously open project source files are closed/disappeared;

if after that I configure/set c++ kit, open some source files ... close and open vscode ... all of the above is repeated again and again.

BTW: I tried to use command-line parameter --user-data-dir just to avoid manual step "copy "/data" subfolder from the previous version" ... but looks like this doesn't work as expected ... extensions and configuration are not available

Regards,
Any help would be appreciated

This issue has been closed automatically because it needs more information and has not had recent activity. See also our issue reporting guidelines.

Happy Coding!


Question: Update rules for author auto-notifications?

@jleighherzog That's exactly the confusion Ryan was talking about—this ticket is only about notifications sent by the system to journals and authors, so we have used the wording "Status change notification was sent to [recipient]" as Ryan suggested

Update rules for author auto-notifications
1) When a curator edits a data submission, regardless of status, upon rebuilding/refreshing I believe the author receives an automatic confirmation email.

For example, this submission (10.7272/Q6Q23XH9) was edited multiple times by a curator in a matter of minutes. I believe that the author received email notifications each time the submission returned to PPR status after being edited. Visit SF case #18031.

Is there a way to eliminate unnecessary auto-emails from being sent to the author? Specifically those notifications that are sent after curators have edited a submission? Perhaps create a rule based on operator? It might be helpful to view a list of rules set up for the auto-emails.

2) Additionally, would it be possible to capture all auto-notifications sent in the Activity Log? It would be helpful to view all actions in the history for each dataset.

I have pulled out task 2 into its own ticket, #2153, as it's a different area of the project and might need discussion over details.

@jleighherzog I've put in a check that won't send the PPR email notification if the submission has ever gone to PPR before. Is that fine, or is there any reason why, or workflow cycle where, you would want users to get multiple PPR notifications?

@jleighherzog After discussion with Ryan and Scott, we are confused about the workflow that caused this issue and its actual frequency (see commentary in URL We think that it seems like an unusual set of circumstances. We'd like more information about it, maybe we can have a call in the new year?

Record email notifications in the Activity Log
Separated from #2083, suggested by @jleighherzog

@ahamelers : Before this is completed, do you need more specifics?

@jleighherzog I've put in code to add a short message each time an email is sent to the author or the journal, like

or

Do you think this will be sufficient? Phrasing okay? I thought we might need to ask questions about when to add the note to the curation activities, but it ended up being pretty easy to add it whenever there is an email sent out that's associated with a specific submission. 😄

To avoid possible confusion, I would reword as "Status change notification was sent to journal"

@ahamelers : This might need a call to discuss too. "Status change notification was sent to journal" doesn't equal "Journal sent status change notification" -- those seem to be different communications to different parties. Let's include this in our call re: #2083?


Question: Need to change loggers location.

I have currently solve this issue by granting access to local user access by:
`chmod +x ngrok_logger.txt` and one more file so i don't get this error for ttgo logger by
`chmod +x logs.txt`

Need to change loggers location.
As we have added logger's functionality, but these are saving logs in the same directory of code so whenever we pull latest code, we need to either stash or checkout, so I have two solutions:

- Change the default location of logger (preferable)
- adds these loggers to git ignore so they won't affect any more.

@Amenawon Please have a look at this issue #71 and then you need to work on staging branch I recently updated the code, feel free to ask for more details if you need and can you please provide any estimate, or should I take care of this issue? as it's not a big deal for me and we must have to solve #71.

Currently I have solved this issue by granting local user access by:
`chmod +x ngrok_logger.txt`

Permission issue on logger file
Today i got

Image

 error.


Question: `ctrl-backspace` not working after swapping the `fn` and `ctrl` keys?

Those above are correct, `fn` is usually handled in firmware so you will need to change it to another key like `control` in your bios before it can be detected and remapped by the OS.

`ctrl-backspace` not working after swapping the `fn` and `ctrl` keys
Hey, I have an Asahi linux installed on an M1. I am using the below config to swap the `fn` and `ctrl` keys:

[ids]
*

[main]
fn = layer(control)
leftcontrol = fn

2 observations I had after applying the above config:
1. `ctrl-backspace` no longer deletes one word to the left.
2. In a text field, `ctrl-shift-left` selects all characters until the start of the text field instead of just selecting up to the start of the current line.

I understand that I can play a little with macros, but I'd really like to use the native functionality to preserve the contextual behaviors of these keybindings; i.e. they should natively behave differently in vim than in a Firefox window.

Shouldn’t `Ctrl–Shift–Left` select only to the beginning of the current word? Anyway, to me this would indicate your fn key still submits fn, because `fn–Left` submits HOME on some hardware, which goes to the beginning of the line and Shift selects. Use the keyd monitor functionality to find out what the output really is.

@luetage It's swapped already, and `fn` acts as `ctrl`. It's the 2 observations that I'm concerned to report. As for #401, it holds true for certain OEMs like Lenovo. However, for Apple, I think that's not available unfortunately except from within MacOS itself (the first thing I did after getting my M1 😃).

@ElSamhaa Can you test this too and verify you're seeing what I'm seeing?
- #comment your rules (/etc/keyd/default.conf)
- sudo keyd -m
- press (and hold) the `fn` key
- press `backspace`
- press `left`
- release `fn`

I get the following output ("delete" instead of "backspace", "home" instead of "left"), which would indicate that the macbook is not sending the original keys for backspace and arrow keys when `fn` is held down (I'm using Fedora Asahi Remix on an M1 MacBook Air)

keyd virtual keyboard 0fac:0ade fn down
keyd virtual keyboard 0fac:0ade delete down
keyd virtual keyboard 0fac:0ade delete up
keyd virtual keyboard 0fac:0ade home down
keyd virtual keyboard 0fac:0ade home up
keyd virtual keyboard 0fac:0ade fn up

@rxhfcy yep, exactly the same with me. I'm on the fedora Asahi remix too btw.

@ElSamhaa great, so this issue can be closed then (not a keyd bug but a feature of the Asahi Linux kernel)

PS. About your earlier comment:

Luckily, that's not true. Here's a pretty easy way to swap `fn` and `control` in Fedora Asahi Remix (permanently, you can later undo this by using `hid_apple.swap_fn_leftctrl=0` instead of `hid_apple.swap_fn_leftctrl=1` if you want):

shell
sudo grubby --update-kernel=ALL --args="hid_apple.swap_fn_leftctrl=1"

...and then reboot. Or if you want to see the effect immediately, you can do this too (not permanent):

shell
echo 0 | sudo tee /sys/module/hid_apple/parameters/swap_fn_leftctrl

Swap fn and leftcontrol on Thinkpad
Is it possible to swap Fn and Control keys on a Thinkpad?

`keyd -m` says that Fn key is `wakeup`, so I did this:

wakeup = leftcontrol
leftcontrol = wakeup

Which makes both `keyd` and `xev` recognize Fn key as Left Control.

However, Fn key still does not function as Control and works with F1-12 keys as before the swap. And Control functions neither as control, nor as Fn key.

I assume this key works on a firmware level and it's not possible to swap them on OS level, right?

If the `Fn` key cannot be detected by the `keyd monitor`, then it is an area of hardware or firmware and should not be subject to control by keyd.

The wakeup keysym generated by the fn key cannot be used as a modifier. The BIOS handles that key and if you use it as a modifier, the BIOS will intercept before keyd gets it.


Question: Is it compatible with Expo 50?

This wasn't overlooked, this was the reason why I pointed out old config-plugins versions in my initial response. The ones with the correct version for this SDK version has been updated to use the correct AppDelegate. Older versions are not.

I'm confused about your contribution to the thread. Were you having the same or a similar issue?

Is it compatible with Expo 50?
### Prerequisites

- [X] I checked the documentation and FAQ without finding a solution
- [X] I checked to make sure that this issue has not already been filed
- [X] I'm sure that question is related to the library itself and not Bluetooth Low Energy or Classic in general. If that so, please post your question on StackOverflow.
- [X] I'm running the latest version

### Question

Hello,

I am using expo 50.

"dependencies": {
 "expo": "~50.0.14",
...

When I tried to run `npx expo install @config-plugins/react-native-ble-plx`, I got the following error:

› Installing 1 other package using npm

npm ERR! code ERESOLVE
npm ERR! ERESOLVE unable to resolve dependency tree
npm ERR!
npm ERR! While resolving: expo-ble@1.0.0
npm ERR! Found: expo@50.0.17
npm ERR! node_modules/expo
npm ERR! expo@"~50.0.14" from the root project
npm ERR!
npm ERR! Could not resolve dependency:
npm ERR! peer expo@"^49" from @config-plugins/react-native-ble-plx@7.0.0
npm ERR! node_modules/@config-plugins/react-native-ble-plx
npm ERR! @config-plugins/react-native-ble-plx@"*" from the root project
npm ERR!
npm ERR! Fix the upstream dependency conflict, or retry
npm ERR! this command with --force or --legacy-peer-deps
npm ERR! to accept an incorrect (and potentially broken) dependency resolution.
...

I can see - it is compatible with Expo 49 in the readme, but I can't install the package with Expo 50. Isn't it compatible with 50?

Thank you,

Hi @Santa0727
We checked compatibility with expo 50 and found no problems. In issue #1176 ( URL is an example using expo 50.

This bugs me 😅

@dominik-czupryna-withintent Do you think it can somehow added anyways?

I was able to successfully make a build 10 days ago with `"expo": "~50.0.4",`

When I want to build my app (`"expo": "~50.0.17",`) with `eas`, I encounter this error URL

Error: [ios.dangerous]: withIosDangerousBaseMod: Unrecognised filter type - 127

CleanShot 2024-05-01 at 09 52 26@2x

`npx expo prebuild` runs fine on my local machine.

Here are my investigations with `yarn why @expo/config-plugins`

yarn why @expo/config-plugins
yarn why v1.22.21
[1/4] 🤔 Why do we have the module "@expo/config-plugins"...?
[2/4] 🚚 Initialising dependency graph...
[3/4] 🔍 Finding dependency...
[4/4] 🚡 Calculating file sizes...
=> Found "@expo/config-plugins@7.9.1"
info Has been hoisted to "@expo/config-plugins"
info Reasons this module exists
 - Hoisted from "expo#@expo#config-plugins"
 - Hoisted from "react-native-ble-plx#@expo#config-plugins" Found "expo-updates#@expo/config-plugins@7.8.4"
info This module exists because "expo-updates" depends on it.
info Disk size without dependencies: "1.73MB"
info Disk size with unique dependencies: "3.82MB"
info Disk size with transitive dependencies: "8.32MB"
info Number of shared dependencies: 60
✨ Done in 0.24s.

On my `app.config.ts` I added this

typescript
plugins: [
 "expo-router",
 [
 "react-native-ble-plx",
 {
 isBackgroundEnabled: true,
 modes: ["peripheral", "central"],
 bluetoothAlwaysPermission:
 "Allow $(PRODUCT_NAME) to detect your bluetooth device"
 }
 ],

@Santa0727
Can you please let us know if `eas build --platform=ios` is working on your side?

Hi @flexbox,
Thank you for your response.
The build command works on my side, and it builds ipa file.

I've used expo managed workflow on my local. Is it impossible to use without ejecting expo?

@flexbox
I mean you shouldn't install URL because react-native-ble-plx is shipped with an Expo config plugin on its own.

@Santa0727
You don't need to eject expo, you just need to prebuild the native code because without using native code this package won't work.

Hi, I am not sure now, where should we add these configurations ?
{
 isBackgroundEnabled: true,
 modes: ["peripheral", "central"],
 bluetoothAlwaysPermission:
 "Allow $(PRODUCT_NAME) to detect your bluetooth device"
}

@kennyyen
in your `app.json` or `app.config.js` URL

json
{
 "expo": {
 "plugins":
 [
 "react-native-ble-plx", // URL
 {
 isBackgroundEnabled: true,
 modes: ["peripheral", "central"],
 bluetoothAlwaysPermission: "Allow $(PRODUCT_NAME) to detect your controller with bluetooth"
 }
 ],
 }
}

Thank you for the response, I did tried to put them in my app.json, and when I do
`npx expo install react-native-ble-plx`
I get this

Skipping config plugin check: Plugin is an unexpected type: object

Thank you for the clarification ! It works just fine as before !

@kennyyen great news!
If your problem is solved can you please close the issue?
It sparks joy for maintainers ✨

[ios.dangerous]: withIosDangerousBaseMod: Could not locate a valid AppDelegate at root
### Summary

Running
`expo prebuild`
or
`expo eject`

### Environment

expo-env-info 1.0.3 environment info:
 System:
 OS: macOS 12.3.1
 Shell: 5.8 - /bin/zsh
 Binaries:
 Node: 16.13.1 - /usr/local/bin/node
 Yarn: 1.22.18 - /usr/local/bin/yarn
 npm: 8.1.2 - /usr/local/bin/npm
 Managers:
 CocoaPods: 1.11.3 - /usr/local/bin/pod
 SDKs:
 iOS SDK:
 Platforms: DriverKit 21.4, iOS 15.5, macOS 12.3, tvOS 15.4, watchOS 8.5
 IDEs:
 Xcode: 13.4/13F17a - /usr/bin/xcodebuild
 npmPackages:
 expo: ~45.0.3 => 45.0.4
 react: 17.0.2 => 17.0.2
 react-dom: 17.0.2 => 17.0.2
 react-native: 0.68.2 => 0.68.2
 react-native-web: 0.17.7 => 0.17.7
 npmGlobalPackages:
 eas-cli: 0.41.1
 expo-cli: 5.0.1
 Expo Workflow: bare

### Please specify your device/emulator/simulator platform, model and version

mac

### Error output

_No response_

### Reproducible demo or steps to reproduce from a blank project

-

I just had this happen to me after ejecting from a fresh Expo app. `expo init my-app` then `yarn eject`.

Upgrading `expo-cli` from `4.12.10` to the latest (`5.4.6`) resolved it for me.

I'm getting the same issue on `expo-cli` `6.0.2` when I run `expo prebuild --clean`

@kbrandwijk @trevordunn @solarisn @Jigalag @hasunperera
Same issue here on expo-cli 6.0.5

expo prebuild --clean
Warning! Your git working tree is dirty.
It's recommended to commit all your changes before proceeding, so you can revert the changes made by this command if necessary.

✔ Would you like to proceed? … yes

✔ Cleared android, ios code
✔ Created native projects | gitignore already synced
⚠ Skipped Metro config updates:
› Existing Metro config found; not overwriting.
› You will need to extend the default @expo/metro-config in your Metro config.
 Learn more: URL

✔ Updated package.json and added index.js entry point for iOS and Android
y🧶 Using Yarn to install packages. Pass --npm to use npm instead.
✔ Installed JavaScript dependencies 1557ms
✖ Config sync failed
[ios.dangerous]: withIosDangerousBaseMod: Could not locate a valid AppDelegate at root: "/Users/XXX/Documents/projects/project"
Please report this as an issue on URL
UnexpectedError: [ios.dangerous]: withIosDangerousBaseMod: Could not locate a valid AppDelegate at root: "/Users/XXX/Documents/projects/project"
Please report this as an issue on URL
 at getAppDelegateFilePath (/Users/XXX/Documents/projects/project/node_modules/expo-screen-orientation/node_modules/@expo/config-plugins/src/ios/Paths.ts:26:11)
 at Object.getAppDelegate (/Users/XXX/Documents/projects/project/node_modules/expo-screen-orientation/node_modules/@expo/config-plugins/src/ios/Paths.ts:63:20)
 at /Users/XXX/Documents/projects/project/node_modules/expo-screen-orientation/plugin/build/withScreenOrientation.js:39:63
 at action (/Users/XXX/Documents/projects/project/node_modules/expo-screen-orientation/node_modules/@expo/config-plugins/src/plugins/withMod.ts:214:29)
 at interceptingMod (/Users/XXX/Documents/projects/project/node_modules/expo-screen-orientation/node_modules/@expo/config-plugins/src/plugins/withMod.ts:100:27)
 at action (/usr/local/lib/node_modules/expo-cli/node_modules/@expo/config-plugins/src/plugins/withMod.ts:228:14)
 at interceptingMod (/usr/local/lib/node_modules/expo-cli/node_modules/@expo/config-plugins/src/plugins/withMod.ts:108:21)
 at interceptingMod (/usr/local/lib/node_modules/expo-cli/node_modules/@expo/config-plugins/src/plugins/withMod.ts:108:21)
 at action (/usr/local/lib/node_modules/expo-cli/node_modules/@expo/config-plugins/src/plugins/createBaseMod.ts:82:21)
 at interceptingMod (/usr/local/lib/node_modules/expo-cli/node_modules/@expo/config-plugins/src/plugins/withMod.ts:108:21)

@kbrandwijk @trevordunn @solarisn @Jigalag @hasunperera Any update on this? we are stuck here.

@kbrandwijk yes, but it was causing the same issue, but I was able to fix the error by following these steps, not sure which step fixed it though,
1.) Upgraded MacOS to the latest version (MacOS Monterey 12.6)
2.) Completely removed Xcode and all the other files
3.) Restarted the machine
4.) Installed latest Xcode 14.0 from app store
5.) Installed other dependencies from the Xcode app (Xcode 14.0 command line tools)
6.) Upgraded expo-cli to latest (version 6.0.5)
6.) Restarted the machine
After following these steps, above error disappeared and was able to run the expo build for IOS successfully. (expo run:ios) @trevordunn @solarisn @Jigalag

Thanks @hasunperera I followed all your steps & I was still getting this error, so I followed this debugging thread (below) from @kbrandwijk.

I had 3 packages pulling in v4.x (old) @expo/config-plugins dependencies — namely expo-cli, eas-cli, and expo. I removed expo-cli & eas-cli from my dependencies since they weren’t strictly necessary (I can just use npx to run each expo & eas in CI so that just left expo.

However I wasn’t experiencing the same error when the project was built with eas. Which was curious. Something Brett Vatne mentioned in some other thread led me to the hypothesis that it could be related to my local node version vs the version I’d specified in eas.json (16.13.0 ) vs my local 16.10.x. I rm -rf’d my yarn.lock & node_modules, set my local nodenv to 16.13.0, reinstalled my dependencies, said a prayer, and ran `expo doctor` and voila, no more @expo/config-plugins version mismatch! Ran `expo prebuild` and no more error.

I just noticed the AppDelegate filename generated with expo prebuild is `AppDelegate.mm` rather than `AppDelegate.m`. I am using Expo 47

Oh I see. Thanks for this insight. I was was making an observation and I had a similar issue with expo prebuild failing.


Question: (JP) Yomitan only scanning the last word in a sentence?

Scan resolution isnt an issue here since it's forcibly turned off when switching to Japanese.

(JP) Yomitan only scanning the last word in a sentence
The last day my Japanese yomitan setup hasn´t been working, and I can only scan the last word or kana of a sentence. Spanish yomitan settings are working fine, but for some reason I cannot scan the majority of words in Japanese. All permissions are given, this is a brand new issue. Image shows that yomitan isn´t completely turned off, but when I scan a sentence and come to the last word it will usually scan it.

To be clear this is not a bug, its a setting in yomitan which I cannot figure out where or what is causing it.

This is almost certainly #1861. You probably need to set `Dictionary search resolution` to `Letter`.

Japanese Language Mode Fails to Automatically Set translation.searchResolution to "letter"
**Description**
When switching the target language to Japanese in Yomitan, the extension does not automatically update the settings `translation.searchResolution` to `"letter"`. Instead, they remain set to `"word"`, which is the default for languages like English. This causes issues with Japanese word segmentation, as the `"word"` mode fails to split compound terms or phrases correctly, leading to failed dictionary lookups.

**Steps to Reproduce**
1. Set Yomitan's primary language to English.
2. Confirm that `translation.searchResolution` are set to `"word"` in the settings.
3. Switch the target language to Japanese.
4. Observe that the settings remain `"word"` instead of automatically switching to `"letter"`.

**Expected Behavior**
When Japanese is selected as the target language, Yomitan should automatically set:
- `translation.searchResolution = "letter"`

**Browser Version**
any

**Yomitan Version**
latest

**Additional Notes**
- Manually changing the settings to `"letter"` resolves the issue, but this should happen automatically when switching to Japanese.
- Languages like Japanese often require `"letter"` mode for granular parsing, while `"word"` works better for spaced languages (e.g., English).


Question: EatingWell zimit attempts fail?

You are right, sorry 😉

But this still needs to be fixed, and this is still very aggressive (I now know how to prevent a whole company/campus from accessing some websites, just do one HEAD request with `python-requests` UA from their public web browsing IP ...)

EatingWell zimit attempts fail
I double-checked the URL and zimit gave a failure on two attempts.

I was using the default options / settings.

URL: URL

Thank you. I appreciate what you are doing.

"Do you know if the site is "protected" by a CDN or something like that?"
I do not know - I had to lookup "CDN".

Each time, Zimit took a long time working on the site before issuing the
Failure.

Dave

On Mon, Oct 16, 2023 at 9:14 AM benoit74 ***@***.***> wrote:

Hello,

I confirm there is an issue most probably linked to some form of protection against denial of service on the website, and this prevents us from proceeding with the content retrieval.

I tested on a server and on my desktop and got the following result:
- I get a systematic error (HTTP 405) on my server machine (even with simple commands like curl, even when I pass a proper user agent)
- it works well on my desktop

I suspect this is linked to some kind of ASN / IP filtering.

I will confirm with the team what we usually do in such a situation.

It is indeed worse than expected. It does not work anymore on my desktop machine. Looks like protection is pretty aggressive.

Ok, Thanks for looking into this for me.

I appreciate your time.

Dave

On Thu, Oct 19, 2023 at 1:58 AM benoit74 ***@***.***> wrote:

I diagnosed this a bit further.

This morning I can request the website via curl again, so clearly there is some kind of active protection.

I achieved to start the website crawl (which is the first phase before creating the ZIM) successfully by passing a User Agent.

@DaveD5501 : could you please try again to request the ZIM on youzim.it and use the "advanced option".

image

In the option named "User Agent", input a valid User Agent from a Browser. You might for instance use mine which is "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/17.0 Safari/605.1.15" (without the quotes).

image

This should be enough to make the crawl start successfully, I will monitor this, should it fail I will keep you informed here.

I tried it again using your User Agent setting. Rather quickly, I got a
failure "Youzim.it task 84262 failed".

Thanks for your efforts.

Dave

On Fri, Oct 20, 2023 at 1:25 AM benoit74 ***@***.***> wrote:

I'm very sorry about that, there is a bug in handling parameters values with spaces.

We will have to fix this before you can proceed, you can track this upstream here: URL

Oh, no, I'm wrong, parameter without quotes is processed properly.

It is just that the User-Agent value is still not working. I will have a look at it on Monday, I suspect I already know the reason.

Ok, thanks for the update.

On Fri, Oct 20, 2023 at 1:27 PM benoit74 ***@***.***> wrote:

By fixing #226, I finally got the real issue, the server does not like our check of the root URL:

failed to connect to URL 406 Client Error: Not Acceptable for url: URL

Fixing #227 will solve the issue.

While doing this, we may also consider making a `GET` request, stopping after the first bytes received. Some web servers don't implement HEAD and the scrape would fail for an invalid reason.

@DaveD5501 now that code has been merged, could you please try to request the ZIM again (no additional parameter needed anymore, just provide the URL)? I tried to start the task and it worked well, not sure it will complete successfully but at least it made much more progress than before.

Thank you for your patience (and thank you for helping us fix all these issues, we made a good progress)

Success! I have my ZIM. I clicked around enough to verify the results.

It took 12 hours and finished overnight. I watched it wait for a slot for several hours - so the long time to complete might have been due to slot availability.

THANK YOU very much.

Great it finally worked, and thank you again for reporting this and for your patience, it helped us solve an issue probably faced by other users.

And btw, yes, the 12 hours is just "bad luck". There was other requests in the pipe, and lot of them also took significant time to complete ; you could request it down it would start immediately to process ^^

Pass same User-Agent to check_url and crawler
As of today, at startup zimit is:
- checking the URL with check_url, which uses Python requests
 - no useragent is specified, so default one is used, e.g. `python-requests/2.31.0`
- starting browsertrix crawler
 - either browsertrix default user agent or the one specified to zimit through CLI flags is used

In all cases, the target server is hence receiving two requests from the same IP in few seconds from two different user agents.

This is not a good thing, and might even be the cause of #224.

We should use the same User-Agent for both situations.

That's very unlikely. This is an isolated **single HEAD request** that happens prior to the crawler staring and then emitting as much requests as it can, using an odd User-Agent.
Keep in mind that an enormous share of Internet users use a shared IP address with their colleagues, household, or ther devices (phone), etc. I'd eat my arm if it was the reason for #224 😅

Absolutely.

I hope you are hungry, because I reproduced the issue 🤣

406

And then from this point, any requests (curl, python, whatever) from the same machine (IP, I imagine) fails with 405 errors.

From a different machine:

200

200

Tada ... (yes, I hate this behavior, but I suspect this might even be the root cause of some situations of #185)

😨 Not so fast, the problem is not the one I quoted (ie. different UA from same IP) but the fact that any request with a `python-requests/` prefixed UA blacklists the IP for some time

py
In [1]: import requests
 ...: resp = requests.head(" URL headers={"User-Agent": "Mozilla/5.0 (Windows NT 6.1; Win64; x64; rv:47.0) Gecko/20100101 Firefox/47.0"})
 ...: print(resp.status_code)
 ...: resp = requests.get(" URL headers={"User-Agent": "Mozilla/5.0 (Macintosh; Intel Mac OS X x.y; rv:42.0) Gecko/20100101 Firefox/42.0"})
 ...: print(resp.status_code)
200
200

In [2]:


Question: More about key reuse?

The RFC's for the ED448 and X25519 forbid use of the contexts, so doing the above violates our principal of using black-box implementations that already exist today.

More about key reuse
So, I think I have a use case for key reuse for composite signatures. To allow the use of the native and alternative private and public keys of Catalyst (hssss) certificates to generate composite signatures. Composite private and public keys for the certificate wouldn't exist in the wild, but the native and alternative private keys of a certificate could be used to generate a composite signature.

Due to the flexibility of these certificates, this means that a signature could be generated corresponding to the native key, the alternative key, or a composite of the native+alternative key.

Some of you authors already know that I'm not generally in favour of Catalyst certificates, but there are those out there who still want to use them, so I'm coming at this from a harm-reduction perspective. And also from a forward-looking perspective, this draft is setting the standard for composite signatures. It would be easier if future composite signatures, for example PQ/PQ, could use the framework of this draft (and implementation) to define other composites (obviously with specific security considerations for the component algorithms involved). At that point, both component algorithms may take a context and key reuse could be done safely.

To that end, I wonder if it would be worth considering enabling key reuse of the component algorithms which take a context parameter by changing the context from:

ctx = Domain

to:

ctx = Domain || Hash(PublicKey1 || PublicKey2)

This would provide domain separation for a key, ML-DSA for example, to be used standalone or as part of multiple composite keys (even multiple composite keys with the same Domain, because the key hash achieves the separation). I don't claim it's a good idea to reuse a key in multiple composite keys with the same Domain, just that this idea would protect the key if that happened.

I'm interested in your opinions.

As Scott says elsewhere, 'belt and suspenders` but this is belt and suspenders and superglue.

My first thought is what if the signer doesn't have access to the public key? My second thought is this works great to obtain what you want for ML-DSA or other component algorithms that take a context, but not at all for the ones that don't (which is RSA and most EC combinations except the Edwards ones). So at best in the context of the current draft it would give us separation for 1 of the components, but not the others, which would foil our security claim of EUF-CMA as long as one of them is EUF-CMA. My third thought is that we had thought about this property a few years ago (For instance, it would work with multiple independent certs as well), and there probably are lots of interesting use cases where keys could be joined to make a composite signature. After lots of discussions it seems this property wasn't desired in lieu of the security implications it brought to composite.

Perhaps for combinations where a context string is used in BOTH algorithms this could make sense... We had talked about using context strings for other components that could support it. See URL but it was mostly decided against because the most compatible way is to use no context String...

So in summary for cases where both components take contexts, I would say it is interesting. For cases where some components don't take a context String, i don't think we want to go there.

Here is someone trying to combine separate keys using the composite signature construct. Check out IPSECME on Monday to see/influence how that goes.
 URL

Regarding your second thought, yes it protects ML-DSA but not the trad key. My point is if someone is going to end up doing this anyways, better to protect only the ML-DSA key than neither key.

Just looking at this again. Dan, are you suggesting that if we made this change:

ctx = Domain

to:

ctx = Domain || Hash(PublicKey1 || PublicKey2)

We should allow re-use of the ML-DSA key (or any other context-supporting algorithm) in some other context?

An attacker could still construct the above ctx and pass that to a regular ML-DSA verifier (with context) and get the stripped signature validated. We recently added a prefix to mitigate this issue in traditional verifiers, #86 but I suppose it could be used by a pure ML-DSA verifier to determine if it is coming from a composite. Then a setting whether a composite validation is being attempted or not would be needed.

If the attacker can control the context used by a signer or verifier then the context parameter has no value and we shouldn't use it anywhere. Wherever a signature context is used, it should be a value entirely controlled by the signer or a protocol-specific value.

The extra context data doesn't have to be derived from the public key, it just has to be something unique to the composite keypair. It could even be some random bytes that is encoded into the composite private and public key formats so both signer and verifier have the same value. The important thing is that it is unique to a single composite keypair.

I am asking:
- Do you think, despite key reuse being forbidden, people will try to do it? I think that fact that draft-hu-ipsecme-pqt-hybrid-auth/ already exists suggests yes.
- If we think people will try to do key reuse, should this draft try to protect them as best it can, despite having told them not to do it?
- Alternately, if we have a way to do composites in a way that allows key reuse, should we?

My idea above would not protect ECDSA/RSA from key reuse, but this comment from a different issue is interesting: URL but was never addressed. It claims to give strong non-separability, although if the component keys were mixed and matched into different composites I think you could still get some separability-adjacent issues. If you took that idea and used a unique context per pair of composite keys (public key hashes, or random bytes encoded in the private and public key) then I think you could fix the mix-and-match problem.

I'm consolidated the idea in #64 since it was related. I think it is an interesting idea to add context separation and the lower algorithm level for ALL algorithms. Today we only add the lower level context separation for ML-DSA... I reference Illari's idea from URL as well.

I've added the comments for #86 into #64. I think we should still forbid Key Reuse, but we could implement your idea which was mirror in URL which I've copied to #64. Essentially making use of context in all algorithms where it can be used, or wrapping algorithms that don't support context with a SHAKE(M',ctx=composite-sigs). I think I want to bring this to the mailing list, but I do think it is an interesting idea...

A couple reasons against it:
- I know some crypto libraries that don't support SHAKE which would now be required to add support for SHAKE BEFORE they could support composite
- Many crypto libraries don't support the context in Ed448 or Ed25519

Hi Dan,

We have discussed this numerous times as an authors group. We keep coming back to the conclusions
- Timing issue
 - we don't want to further delay the document by needing further cryptographic review.
- Usability issues:
 - needing to have the public keys at signing time
 - encoding of the public keys needs to be identical on both sides, which may not always be the case.

The above are mitigated by the fact that re-use is forbidden. This seems like a good middle ground compromise. This is also mitigated by Falco's suggestion of the prefix which if used by a traditional verifier would detect that it should have been a composite signature and not a single signature.

In the context of an HSM, the above idea doesn't work as the signer doesn't even have access to the private key... All they have is a reference to the private key. Therefore any artifacts added to the private key will not be available, meaning the full private key is not available.

I discussed this with Mike Ounsworth today, and probably the best option is to make use of the application specific context if you want to use composites with the Catalyst certificate. So if we add something like the following to the draft, does this satisfy your request Dan?

"In the event that an application wishes to use two separate keys (for example from two single-algorithm certificates) and use them to construct a single Composite Signature, then it is RECOMMENDED to provide a composite ctx to prevent this signature from being validatable under a composite key made up of the same two component keys.

For example, an application or protocol called Foobar that wishes to do this might invoke the Composite algorithm as:
Composite-ML-DSA.Sign( (sk1, sk2), M ctx="Foobar-dual-cert-sig", PH)"

I don't think that suggestion would help since the application context being provided is only used to create the Composite-ML-DSA M'. So a component signature can still be separated from the composite signature to provide a valid single-algorithm signature for M'. The suggestion would help when using the same composite keypair in two different protocols, but it wouldn't help when using the composite keypair and reusing one of the components.

Hi Dan,

The suggestion was for usages outside of this specification like in the catalyst certificate as you mention. Of course anyone that knows the application specific context could re-construct the message:

M' = prefix | domain | ctx(len) | ctx | HashOID | PH (r | Message)

As of this week, they would also need the r from the full composite signature to make the construction before they could verify. At that point if they are doing all this message work to verify one component of the composite signature, they are basically purposefully doing a partial composite validation but calling it a single validation.

Would putting the application context or possibly r into the ML_DSA context help? Lets think about it.

Thanks Dan,

I agree, adding the r into the context gives the attacker some control over the context string, which could be manipulated.

So we are left with this:

The r does help with some subset of key reuse, but not all cases. It solves we call the "mixed-key forgery attack": Take two composite keys `(mldsaPK1, tradPK1)` and `(mldsaPK2, tradPK2)` which do not share any key material and have them produce signatures `(r1, mldsaSig1, tradSig1)` and `(r2, mldsaSig2, tradSig2)` respectively over the same message `M`. Consider whether it is possible to construct a forgery by swapping components and presenting `(r, mldsaSig1, tradSig2)` that verifies under a forged public key `(mldsaPK1, tradPK2)`. This forgery attack is blocked by the randomizer `r` so long as `r1 != r2`.

Also if we have this more conventional key re-use scenario

Keya = MldsaA+TradA1
Keyb = MldsaA+TradA2

Any signature produced by Key1 or Key2 would be of the form
(r1, siga1,siga2) -> siga
(r2, sigb1, sigb2) ->sigb

M1 = prefix | domain | ctx(len) | ctx | HashOID | r1 | PH (r1 | Message)
M2 = prefix | domain | ctx(len) | ctx | HashOID | r2 | PH (r2 | Message)

If you try to verify siga using the sigb key (as a composite), it will fail thanks to the traditional component, but that doesn't prevent key re-use:
You need to compute:
M1 = prefix | domain | ctx(len) | ctx | HashOID | r1 | PH (r1 | Message)
Keyb.mldsa would be able to replace Keyb.mldsa (its the same key and ctx domain), but the traditional component would still cause an overall failure.

- if r1 was passed down into the ML-DSA ctx, so that MLDSA.ctx = domain | r1 then:

MLDSA.Verify(Keyb.MldsaA, M1,ctx=domain | r1) = MLDSA.Verify(Keya.MldsaA, M1, ctx=domain | r1)

So it wouldn't help at all.

- if Hash (Keya) is passed down into the ML-DSA ctx, so that MLDSA.ctx = domain | Hash (Keya) - the original request:

k = Hash(Keya)
MLDSA.Verify(Keyb.MldsaA, M1, ctx=domain | k) != MLDSA.Verify(Keya.MldsaA, M1, ctx=domain | Hash(Keyb) )

This works because you can do a check on the full public key being used to verify the signature at that instance. I think the only solution is probably using the Hash of the public key (or some combined aspects of the key itself that can be checked by the honest composite verifier). Thus we have come full circle.

So unfortunately, given the design constraint of not wanting to require access to the public key at signing time, and the fact that the private key components may not always be available in bytes but only in a reference on a HSM, I think we are left with the very strong language against key re-use in the draft.

Should we use the context String for the underlying ED448 and X25519 algorithms
We are using the Domain as a context String for ML-DSA,

We can't use the Domain as the context for RSA or ECDSA as they algorithms don't take a context string.

However, ED448 and X25519 do use a context String and could be used.

That would likely give us SUF security for those algorithm combinations.

Both RFC8410 (EdDSA in X.509) and RFC8419 (EdDSA in CMS) say explicitly that the context string is not used. Since currently X.509 and CMS do not use the context string of EdDSA, then the most backwards compatible thing is to maintain that behaviour for the EdDSA component.

The comments from Illari in this comment: URL would possibly allow context separation to work for all the algorithms. Maybe this is something to put on a slide at 122.

Adding the text here so its not lost:
Alternatively, since RSA and ECDSA actually sign a hash, and everything else relevant has context inputs, one could do something like the follows to get actual strong non-separability (even for RSA and ECDSA):

Pure:

M' = Domain | 0 | len(ctx) | ctx | M

Prehashed:

M' = Domain | 1 | len(ctx) | ctx | HashOID | PH(M)

RSA:

RSA_signhash(cSHAKE256(M', ctx="sig-composite"))

ECDSA:

ECDSA_signhash(bytes2bits_be(cSHAKE256(M', ctx="sig-composite")))

Ed25519:

Ed25519ctx_sign(M', ctx="sig-composite")

Ed448:

Ed448_sign(M', ctx="sig-composite")

ML-DSA:

MLDSA_sign(M', ctx="sig-composite")

Some notes:

The bytes2bits_be() is to ensure that ECDSA interprets the hash in the expected way.
This aligns with the model presented in the meeting.

@johngray-dev is this issue still open, or have we decided to stay with the current design?


Question: Question-How are streams determined from files?

I agree with @pauladkisson that we should break streams. As a note, Streams are not about different types of preprocessing. It is basically stuff that can be processed together in the context of spikeitnerface and raw.io in neo (same sampling_rate, start and duration). The concept is not documented but you can find somedetails here.

How to break depth, channel and whatever else might come?
* I think that depth (moving across planes) is a natural category. Space is there and is not moving anywhere. This should be a thing on its own and be addressed as an index/number. Moreover, this is dimension that the `PhotonSeries` in pynwb accepts and other related modalities take for granted (e.g. brain imaging). I think is a solid well defined concept so it gets a specification.
* Channel is probably better addressed with a string. Sometimes is going to be a number but I also think that the concept of channel is less defined in ophys. In the current conversion that me and @alessandratrapani are working, this is used to describe two different filters (and therefore two molecules) but this does feel less stable and way more ad-hoc than space. Therefore, I think it should get a very (the most?) general type.
* Whatever else might come? I feel that the instinct of @CodyCBakerPhD is right in trying to future proof here. We don't know how ophys will change and we might not want to commit ourselves to some specific categories. In that light, creating a common API which is now called streams to reflect this diversity (channel, modality, preprocessing or whatever else) is a good way of building resilient structure that we can move forward as technology changes. With the exception of deph for the reasons mentioned above, I would be fine with having optical channel dumped in streams along anything else that might come in the future specified as keyword argument with type string. This could be stream as we currently have it, could be modality, mode or any other word that we find and agree is better.

As for this:

MultiImagingExtractor was designed to concatenate ImagingExtractors along the frames axis, but here we want to combine multiple IEs along a separate depth axis.

This is not true:
 URL
But I think it was changed here:
 URL

I agree that we are conflating a tool to concatenate data extractors in time with some internal machinery to assemble extractors from channels, planes or sub-extractors in time. We came to need the latter because some formats provide multiple files and we judged (corrrectly?) that composing in this way is easier than ad-hoc parsing. I think it makes sense separting assembling in time, channels and depth.

The analgous in spikeinterface are concatenation and channel agregation.

Can you point out to where does this show up and also add other wonky stuff / difficulties? I feel this point should be more descriptive of existing problems.

I will add something about the third point later.

Question-How are streams determined from files?
For example, I have a plexon file that contains data grouped into 3 catagories. The wideband, the field potential of the wideband, and the spikes from the wideband. When reading the file and accessing a specific stream, the field potential is grouped into one stream and the wideband as well as the spikes are grouped into another stream. How are these streams determined and is it possible to separate the wideband and spikes into separate streams so I am able to only access one or the other? The wideband and spikes both have a sampling frequency of 40,000 hertz while the field potential has a sampling frequency of 10,000 hertz?

plexon 1 or plexon 2? (plx for pl2 files). The docs explain the general idea here.

But basically if the sampling frequencies are different they have to be different streams. I'm surprised Wideband and spikes are in the same stream. Once you let me know which plexon I'll dig into more :)

I see. yeah that might be a little bit of poor organization on our part. It seems like with the current setup you would just have to do a channel slice to get the correct channel for your purposes instead. Could you do the following and put it here.

python
from neo.rawio import PlexonRawIO

reader = PlexonRawIO(file_name)

reader.parse_header()

print(reader)
print(reader.header)

Once I double check how that displays I could check to see if there is a way to split those into separate streams. I might need a test file though.

That's why I'm having you run the print reader and print reader.header. If we aren't labeling the channels right, then it would be impossible, but often we give the channel's "names" which you could use to do the slice. Could you do the script I posted above so we can see if it is possible or whether we need to patch that right away ?

Spikeinterface defaults to the channel_ids

so above we have ('WB01', '0',), so channel_name 'WB01' is channel_id '0' and channel_index 0. So yes for a `ChannelSliceRecording` in this case you slice from '0' - '15' to only get the WB channels.

It might be helpful if we make the channel_ids more meaningful in this case since we are losing WB vs spike information when going from name -> id which could cause mistakes for end users. I'll wait for some neuroconv people to weigh in, but I think if we can guarantee that the names are unique then we could use the names as the ids as well.

Two things:
1) At spikeinterface level there is an option to change names as ids that should be used for plexon. I think this will make it easier. I will open a PR.
2) @AbhiSwamiUConn I have a question, are you referring to the channels called `SPKC` as spikes? According to a plexon engineer that contribute here recently says that is just a filtered version of the wideband . That's the reason they are the same stream, they have the same dtype, they come from the same file and they have the same sampling rate.

I think that a stream is a very unfortunate name that has already expanded to other parts of the ecosystem and has lead to a lot of confusion. The thing is that `stream` is just an implementation detail of neo. It is the part of the format data that cean be thought as a single block and put into a numpy or memmap array. That's why it is defined by having the same dtype, sampling_frequency, shape and coming from the same file. Nothing more.

The problem is that we expose this implementation detail at the user level interface by making users access the data through streams. Users then get confused thinking that stream must mean something like "a type of qualitatively different data" and in some cases it makes sense (neuropixel has a stream for ap and lf bands for example) but in some others it does not see here my own confusion in the past:

 URL

I don't know if this should be made more prominent in the docs. As I mention it has lead to confusion in others parts of the ecosystem:
 URL

Thanks for the clarification:
In spikeinterface it should be easy if we merge this PR:
 URL

Thanks for clarifying both. Sorry yeah spikes in neo are suppose to be their own thing, but if spkc if just filtered than in Neo parlance it would absolutely be in the same stream.

I agree we should probably find a better explanation of stream for our docs!

OK, now that URL is merged on spikeinterface I wrote the asnwer to your question in #3196 @AbhiSwamiUConn

I think that we can close this @zm711 or maybe you want to leave it open until we write better documentation for this? The thing is that I am not sure the stream is ever exposed on neo directly, right?

Actually, final question @AbhiSwamiUConn. Do you happen to know what are the types of filering that is implemented for the "FP" and "SPKC" channels? It would be good for us to know more about this metadata.

They are exposed. here for example. The stream index gives us the sample rate, the gain, and the offset for the data so we can convert it and is necessary for the user functions. So I think we do need to document it. But stream itself is not exposed, just the indices of the streams. Is that your point?

My question was not well phrased. I guess the core of my question? Do users of neo need to know about streams the same way that users of spikeinterface need to know about streams?

In spikeinterface you need to know the streams to access the data but as I am not a userr of neo I don't know how users interact with the concept of stream and if this type of confusion can arise at the neo level.

Yes.

python
reader = neo.RawIO()
reader.parse_header()

in this example the reader will have the memmap for all streams, but to actually pull out the data you need to specify which stream of data you want. I would actually say that spikeinterface is nicer in that it can be fed with name, whereas all the functions at the rawio level expect the stream index. So you run into the issue that if you don't know the IO it isn't clear how to get what you want

ie how do you get digital data for intan -> you need stream_index = 3. So you basically need to look in the header to see which stream goes where. Again Intan has good names in the header, but it seems like Plexon says streams are Signal 0 and Signal 1 which aren't super helpful.

Yes, we probably should have better signal streams in plexon but I we will need someone who actually has experimental experience with the format to venture in that direction.

Thanks, I wonder if the values of the bandpass are fixed or something that people decide at run time.

Unify Volumetric/MultiPlane Imaging API
# Current Behavior

Right now, the only imaging extractor that supports 3D imaging is BrukerTiff. It does so by taking a list of `SinglePlaneImagingExtractors` and combines them a la `MultiImagingExtractor`. Note that each `SinglePlaneImagingExtractor` is itself a `MultiImagingExtractor` that combines multiple 'chunked' files corresponding to a single depth plane. Each `SinglePlaneImagingExtractor` instance only supports 1 channel and 1 plane, which are passed via `stream_name` during the `__init__`. Each `MultiPlaneImagingExtractor` instance supports 1 channel (but all planes) which is passed via `stream_name` during the `__init__`.

# Issues

- 'streams' are a low-cohesion concept to represent optical channels and depth planes.
 - They are referred to by string names, which can be different in the `MultiPlaneImagingExtractor` (which only needs to specify the channel) and the `SinglePlaneImagingExtractor` (which needs to specify some formatted combination of channel and plane).
 - They are supposed to be analogous to 'AP', 'LF', etc. streams in spikeinterface, but those streams refer to collections of channels that undergo different types of pre-processing. In contrast, each stream as they are implemented in BrukerTiff just refer to 1 channel and 1 plane. There may be a relevant parallel in ophys (ex. functional vs anatomical streams), but specifying a plane/channel combo is not that parallel.
 - optical channels and depth planes are fully independent concepts (1 plane + many channels, 1 channel + many planes, many channels + many planes are all possible) with rich associated metadata that can guide the API: channel name, channel wavelength, plane depth, etc. By lumping these two axes together as 'streams' we lose the ability to interact with them separately.
- Combining multiple `SinglePlaneImagingExtractors` with `MultiImagingExtractor` is a bit hacky and confusing.
 - `MultiImagingExtractor` was designed to concatenate `ImagingExtractors` along the `frames` axis, but here we want to combine multiple `IEs` along a separate depth axis.
 - Using `MultiImagingExtractor` for this purpose leads to wonky stuff like `start_frames = [0, 0, ..., 0]`.
- BrukerTiff specifies channel during the `__init__` but all other IE's specify channel during the call to `get_video`.
 - Not sure which is better, but we should be consistent.

# Proposed Solutions

- Specify channel and plane separately by index (or ID?) and then the extractor handles the logic of how to load that channel/plane combo in their file structure (Bruker, ScanImage, etc.)
- Add a `MultiPlaneImagingExtractor` base class to handle the combination of IE's along the depth axis. Could come with extra metadata properties like `get_plane_depth`.
- We need to specify plane at `__init__` time so that SinglePlaneImagingExtractors can be appropriately combined into MultiPlaneImagingExtractors. So, to keep it consistent we should specify channel also at `__init__` time. This will require refactoring of all the existing IE's but I think it is the most convenient implementation agnostic to file structure. It also simplifies `get_video` to only take `start_frame` and `end_frame`, which I think is a good thing.

### Do you have any interest in helping implement the feature?

Yes.

### Code of Conduct

- [X] I agree to follow this project's Code of Conduct
- [X] Have you ensured this bug was not already reported?

@CodyCBakerPhD @h-mayorquin @weiglszonja @alessandratrapani Please let me know what you all think!

@pauladkisson Thanks for writing this all up, I'll look deeper and leave more comments over next couple days

Not sure which is better, but we should be consistent.

Strongly against having it in the `get_video` call because it leads to the 4D ambiguity; much clearer to have separate objects for each channel if you want to do anything with them side-by-side, which makes it a field specified at `__init__`

So totally on board with that solution

Yes and no - their code will have to be changed, but it was also code that was never tested or used perhaps outside of the `.sbx` case? Which we've also not seen used in quite a while

👍

Yeah, been talking with @h-mayorquin about this lately. It's purposely ambiguous to also allow any other type of future formatting distinction we might need to add while remainig universal across the API

It's also flatter and simpler, though what it really does is shift the complexity of structure to the string itself, but as long as we have a convenient fetcher `get_stream_names` that is less complicated than breaking the approach into separate calls per plane/channel option.

Multiple optic channels found? It shows up as a series of structured strings. Multiple planes found? They show up as another part of the string structure. Want to initialize an extractor? Just specify a single string that matches one of the selections.

Strongly against the ID/index route. Neo supports both at the same time and the ID is the annoying one, the bane of many a stream index issue. Strongly recommend using a human-readable identifier. Less strongly against splitting `stream_name` into `channel_name` + `plane_name` if others think that is worth it compared to my 'single argument is simpler' point above

Thanks for the pointer @weiglszonja. I still would like to see more confusions that arise from this use. This is the only case that we have for Paul's second point.

# Notes from Barcelona 2023:
- Consensus that optical channels should be separate Extractors --> need to refactor old stuff to fit
- 2 Main goals for the repo:
 - Make conversion easier
 - Define unified API for users like SpikeExtractors in e-phys
- Guiding Principle: Focus on how it looks for the user (usu. conversions) and then design the API around that
- specify channel_name and plane_name via string at init
- look out for segments in the future if/when we get data
- numpydoc_validation needs some kind of automatic docstring fixer before it's worth adding


Question: Indexing boolean fields breaks something?

I'm talking to the maintainers of the other Redis OM Clients for different languages to make sure that when we do add booleans to Python there's a common approach where possible.

Indexing boolean fields breaks something
python
class Something(JsonModel):
 name: str
 should_do_something: bool = Field(index=True, default=True)

Migrator().run()

Something(name="test").save()

result:

Something(pk='01H0HYP3C4YFF1MR6JV9MEZSF4', name='test', should_do_something=True)

executing a basic get by pk works fine.

python
Something.get('01H0HYP3C4YFF1MR6JV9MEZSF4')

result:

Something(pk='01H0HYP3C4YFF1MR6JV9MEZSF4', name='test', should_do_something=True)

when running find().all(), the result is empty:

python
In [22]: Something.find().all()
Out [22]: []

If I set the bool value to Index=False, the indexing starts working fine:

python
In [24]: class Something(JsonModel):
 ...: class Meta:
 ...: database = get_redis_instance()
 ...:
 ...: name: str
 ...: should_do_something: bool = Field(index=False, default=True)
 ...:

In [25]: Migrator().run()

In [26]: Something.find().all()
Out[26]:
[Something(pk='01H0HYP3C4YFF1MR6JV9MEZSF4', name='test', should_do_something=True),
 Something(pk='01H0HYGDBJF50GFRYK9MSPA8DQ', name='test 2', should_do_something=True),
 Something(pk='01H0HYB0MKSMV4QWJ7BM5G2DGH', name='test', should_do_something=True)]

I can confirm this as well, that the JSONModel doesn't support indexing boolean fields. Here is my testcase that shows the same thing:

python
"""This file demonstrates that redis_om_python cannot index/search boolean fields"""
from redis_om import Field, JsonModel, Migrator

class MyObjectBroken(JsonModel):
 my_string: str = Field(index=True)
 my_int: int = Field(index=True)
 my_bool: bool = Field(index=True)

class MyObjectWorks(JsonModel):
 my_string: str = Field(index=True)
 my_int: int = Field(index=True)

Migrator().run()

# Add some objects that are broken
object1 = MyObjectBroken(my_string="", my_int=1, my_bool=False)
object1.save()
object2 = MyObjectBroken(my_string="", my_int=1, my_bool=False)
object2.save()

# Add some objects that work
object3 = MyObjectWorks(my_string="", my_int=1)
object3.save()
object4 = MyObjectWorks(my_string="", my_int=1)
object4.save()

objects = MyObjectBroken.find().all()
print(f"Found {len(objects)} MyObjectBroken expected 2")

object_get = MyObjectBroken.get(object1.pk)
print(f"Found {object_get=} MyObjectBroken")

objects = MyObjectWorks.find().all()
print(f"Found {len(objects)} MyObjectWorks expected 2")

object_get = MyObjectWorks.get(object3.pk)
print(f"Found {object_get=} MyObjectWorks")

If you run this, you'll get the following output showing the object with a boolean field doesn't work:

text
$ python boolean_field_search_suppport.py
Found 0 MyObjectBroken expected 2
Found object_get=MyObjectBroken(pk='01H5K85HDADT687Y25Y7BAEWKK', my_string='', my_int=1, my_bool=False) MyObjectBroken
Found 2 MyObjectWorks expected 2
Found object_get=MyObjectWorks(pk='01H5K85HDCCAS49NADBDY39CXQ', my_string='', my_int=1) MyObjectWorks

See URL for where it's defined that booleans aren't supported yet, but seem to be supported at some point?

From my sleuthing, since the `bool` class is implemented as a subclass of the `integer` class, we end up getting a numeric index created:

text
"ft.create" ":__main__.MyObjectBroken:index" "ON" "JSON" "PREFIX" "1" ":__main__.MyObjectBroken:" "SCHEMA" "$.pk" "AS" "pk" "TAG" "SEPARATOR" "|" "$.my_string" "AS" "my_string" "TAG" "SEPARATOR" "|" "$.my_int" "AS" "my_int" "NUMERIC" "$.my_bool" "AS" "my_bool" "NUMERIC"

This is coming from this line:
 URL

From the docs of `FT.CREATE` there isn't native support for boolean types, so it likely has to be mapped to `NUMERIC` for this to work, however it's unclear what's going on under the hood here, but I'm guessing that at least a part of it is related to saving actual RedisJSON boolean fields, but telling RediSearch that it's a numeric field.

I found a workaround for this issue. Since the `bool` class is a subclass of the `int` class, you can declare the model field as `int` and then convert between bool/int as necessary.

Here is the code that I tested on redis-om 0.1.3:

python
"""This file demonstrates that redis_om_python cannot index/search boolean fields"""
from redis_om import Field, JsonModel, Migrator

class MyObjectBroken(JsonModel):
 my_string: str = Field(index=True)
 my_int: int = Field(index=True)
 my_bool: int = Field(index=True) # declare as int, but pass in boolean values works

class MyObjectWorks(JsonModel):
 my_string: str = Field(index=True)
 my_int: int = Field(index=True)

Migrator().run()

# Add some objects that are broken
object1 = MyObjectBroken(my_string="", my_int=1, my_bool=False)
object1.save()
object2 = MyObjectBroken(my_string="", my_int=1, my_bool=True)
object2.save()

# Add some objects that work
object3 = MyObjectWorks(my_string="", my_int=1)
object3.save()
object4 = MyObjectWorks(my_string="", my_int=1)
object4.save()

objects = MyObjectBroken.find().all()
print(f"Found {len(objects)} MyObjectBroken expected 2")

objects = MyObjectBroken.find(MyObjectBroken.my_bool == int(False)).all()
print(f"Found {len(objects)} MyObjectBroken with bool as False expected 1")

object_get = MyObjectBroken.get(object1.pk)
print(f"Found {object_get=} MyObjectBroken")

objects = MyObjectWorks.find().all()
print(f"Found {len(objects)} MyObjectWorks expected 2")

object_get = MyObjectWorks.get(object3.pk)
print(f"Found {object_get=} MyObjectWorks")

Here is the output from it:

text
$ python boolean_field_search_suppport.py
Found 2 MyObjectBroken expected 2
Found 1 MyObjectBroken with bool as False expected 1
Found object_get=MyObjectBroken(pk='01H5KBCJMMCY9Z9SY74C87B20N', my_string='', my_int=1, my_bool=0) MyObjectBroken
Found 2 MyObjectWorks expected 2
Found object_get=MyObjectWorks(pk='01H5KBCJMP4RM26QSSVB77YKTQ', my_string='', my_int=1) MyObjectWorks

Missing support for boolean field or examples of how to do it
I have a model like
`
MyModel(HashModel):
 local_auth: bool
`
When initing or saving I get:
`DataError("Invalid input of type: 'bool'. Convert to a bytes, string, int or float first.")`

Hi there - boolean fields aren't yet supported for JSON or Hash, they'll likely appear in JSON first as boolean is a native type there vs a Redis Hash that understands strings, floats and ints.

@simonprickett , any update on native support for boolean fields?


Question: Pod metrics not shown?

Thanks @SquupS . We'll take a closer look and see how we can improve our metrics display.

Pod metrics not shown
The node metrics are show. However the pod metrics are not shown. Can headlamp display that? the cpu and memory usage along with requests and limits information would greatly help

Here is my output of `k top pods -A`

NAMESPACE NAME CPU(cores) MEMORY(bytes)
default cf-app-78ffc8ff85-wmg7g 3m 30Mi
kube-system calico-kube-controllers-759cd8b574-vtp4k 2m 30Mi
kube-system calico-node-58fdx 24m 100Mi
kube-system coredns-7896dbf49-ktv8n 2m 30Mi
kube-system metrics-server-df8dbf7f5-4jjjp 3m 35Mi
traefik traefik-6d574648c7-pz7vf 1m 79Mi
vscode-ns vscodeapp-8c8687db7-4bjkl 2m 192Mi

Hi! As I know for pod metrics to work you should have prometheus installed inside your k8s cluster

Is this a headlamp requirement? Since pod metrics generally works fine without it as per my snapshot in earlier post above

Yes, headlamp use prometheus plugin for rendering metrics, it search for a prometheus instance inside of the k8s cluster and grab metrics from it.

Hi @manju-rn you can try installing prometheus in Headlamp app by selecting the Apps page from the sidebar. Please check prometheus plugins for more information.

The Kubernetes project currently lacks enough active contributors to adequately respond to all issues.

This bot triages un-triaged issues according to the following rules:
- After 90d of inactivity, `lifecycle/stale` is applied
- After 30d of inactivity since `lifecycle/stale` was applied, `lifecycle/rotten` is applied
- After 30d of inactivity since `lifecycle/rotten` was applied, the issue is closed

You can:
- Mark this issue as fresh with `/remove-lifecycle rotten`
- Close this issue with `/close`
- Offer to help out with [Issue Triage][1]

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle rotten

[1]: URL

I think this issue can be closed because there is an issue open for it here:
- URL

Getting real time mertrics on Pod level for CPU/Memory requests and Limits?
Hi,

is it planned to implement real time metrics, so that the user can see what has been set as request and limit (CPU and memory) per pod and how much is utilized in %?

I think that would be a great addition, if not already present and I missed a switch or so :-)

Best regards

Hi @SquupS . We do show metrics if the metrics-server is installed in the cluster which does show the current utilization. Can you check if this is what you mean?
We're also working on time based charts, powered by Prometheus.

Is there the ability to translate Memory to `Mi` as opposed to the raw value?

@thepaulmacca Can you show a screenshot so I know exactly which part of the UI we're talking about?

@joaquimrocha I'm only on mobile atm sorry, but if it helps it was when looking at HPAs

Sorry for the late reply. So I am thinking of something a "competitor" of yours if offering on Pod level

image

This would really help tracking down resource issues. As of now, I always have to use something like for example k9s to get the current metrics of the pod. Maybe this calrifies a bit where I was originally coming from.

Best regards

@joaquimrocha Hi!
Any updates on this issue?
App is great, but this really useful feature is missing :)

Is this the prometheus plugin? On my local env I see:
image

@JoniJnm Sorry, I wasn't clear, my question was about pods requests/limits info (and maybe % of utilization from real time use).

Agree, this is very useful information.
It's in the Grafana dashboard if you have that installed, but would be simpler for our less experienced users if it were in Headlamp too.
Example from Grafana
image
And from k9s
image

The Kubernetes project currently lacks enough contributors to adequately respond to all issues.

This bot triages un-triaged issues according to the following rules:
- After 90d of inactivity, `lifecycle/stale` is applied
- After 30d of inactivity since `lifecycle/stale` was applied, `lifecycle/rotten` is applied
- After 30d of inactivity since `lifecycle/rotten` was applied, the issue is closed

You can:
- Mark this issue as fresh with `/remove-lifecycle stale`
- Close this issue with `/close`
- Offer to help out with [Issue Triage][1]

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle stale

[1]: URL

pushed up a change for this, lmk if you'd prefer a different look. I figured a tooltip would work best because requests and limits are optional values and the table is already pretty packed with info. here's what it looks like in the context of the whole table on a larger screen:

I'm not sure about this.
As I said earlier there is still requests part and it should be placed somewhere.
From my point of view in pod list the most interesting is the pod limits along with actual usage, for resource requests I would just go to the pod's details (but still make sense show in details requests AND limits).
In other words: main place for requests and limits is the pod's details, but limits duplicated in pods list view.

I would also like to see it in the specific pod view. Bonus points would be to plot it on the prometheus metrics dashboard as well to allow comparing usage to what's tolerated.