Question: Jetbrains Toolbox "Invalid client serial" when using dbus-user=filter?

This has to do with the exception handling of the calling code.
When DBus is not available dbus-java will instantly fail with an `AddressResolvingException` or an `IOException`. When the calling code swallows the exception, nothing will happen (no DBus usage, no blocking).

When blocking certain calls the problem is different. While establishing connection to DBus worked (handshaking, sending Hello etc), specific calls get blocked. In that case the connection (at least in version 4.0.0) gets into some sort of half-disconnected behaviour, which causes the connection to be treated as connected when it actually isn't.
This was fixed in 4.1.0 of dbus-java.

When using 4.1.0 with your setup, the application will only block for about 10-20 seconds due to waiting for method replys which will never be received.

I improved this in the latest 4.1.1 snapshot, so dbus-java will no longer wait for method-replies if the disconnection from the bus was caused by an `IOException` (e.g. of the transport like in this case).

The error message 'invalid client serial' you see is not from dbus-java but from some launcher used to start the jetbrains-toolbox.
I guess that this has something to do with messages not arriving at that application, but usually this does not hurt.

Jetbrains Toolbox "Invalid client serial" when using dbus-user=filter
### Description

Jetbrains Toolbox takes an insane amount of time to load up and display its tray icon when using `--dbus-user=filter`. Unlike a normal run, it displays the following message:

** (process:10596): WARNING **: 21:28:17.026: Invalid client serial

This message originates from xdg-dbus-proxy, in flatpak-proxy.c.

### Steps to Reproduce

1. Download the toolbox app URL
2. Run `firejail --noprofile --dbus-user=proxy --appimage ./jetbrains-toolbox`

### Expected behavior

The app runs and displays a GtkStatusIcon as a fallback for having no access to a StatusNotifierItem tray.

### Actual behavior

It took a solid 5 minutes to start up and do this.

### Behavior without a profile

N/A

### Additional context

I don't have a StatusNotifierItem implementation like snixembed, yet running outside of firejail makes it work. I want it to run with either --dbus-user=none or --dbus-user=proxy, either is fine.

### Environment

- Gentoo amd64
- i3 window manager with polybar

firejail version 0.9.70

Compile time support:
 - always force nonewprivs support is disabled
 - AppArmor support is disabled
 - AppImage support is enabled
 - chroot support is enabled
 - D-BUS proxy support is enabled
 - file transfer support is enabled
 - firetunnel support is disabled
 - IDS support is disabled
 - networking support is enabled
 - output logging is enabled
 - overlayfs support is disabled
 - private-home support is enabled
 - private-cache and tmpfs as user enabled
 - SELinux support is disabled
 - user namespace support is enabled
 - X11 sandboxing support is enabled

### Checklist

- [x] The issues is caused by firejail (i.e. running the program by path (e.g. `/usr/bin/vlc`) "fixes" it).
- [x] I can reproduce the issue without custom modifications (e.g. globals.local).
- [ ] The program has a profile. (If not, request one in ` URL
- [ ] The profile (and redirect profile if exists) hasn't already been fixed upstream.
- [x] I have performed a short search for similar issues (to avoid opening a duplicate).
 - [x] I'm aware of `browser-allow-drm yes`/`browser-disable-u2f no` in `firejail.config` to allow DRM/U2F in browsers.
- [ ] I used `--profile=PROFILENAME` to set the right profile. (Only relevant for AppImages)

### Log

 Output of LC_ALL=C firejail --noprofile --dbus-user=proxy --appimage ./jetbrains-toolbox

Parent pid 10594, child pid 10598

** Warning: dropping all Linux capabilities and setting NO_NEW_PRIVS prctl **

Mounting appimage type 2
Warning: An abstract unix socket for session D-BUS might still be available. Use --net or remove unix from --protocol set.
Child process initialized in 50.05 ms

** (process:10596): WARNING **: 21:28:17.026: Invalid client serial

`--dbus-user=proxy` is not supported. It's `filter`.

---

The fallback code a program uses if an dbus name does not exists is outside of the scope of firejail, consider reporting this to jetbrains then.

If you believe this happens because of an implementation deatil of xdg-dbus-proxy, report it there.

TL;DR: Is this a firejail issue?

I meant --dbus-user=filter, of course. Does --dbus-user=proxy even work?

Anyway, this program uses libappindicator. That's what uses GtkStatusIcon as a fallback.

This problem doesn't happen without firejail, as the fallback works properly in that case, though it's very possible that it's a xdg-dbus-proxy thing.

Yeah, I managed to reproduce it with the following setup:

xdg-dbus-proxy "$DBUS_SESSION_BUS_ADDRESS" dbus --log --proxy &
DBUS_SESSION_BUS_ADDRESS="unix:path=$PWD/dbus" ~/.local/share/JetBrains/Toolbox/bin/jetbrains-toolbox

The library causing the hang turned out to be dbus-java, not libappindicator. The issue is further discussed here, with a potential fix that might appear in future releases: URL

For now, a different workaround to get the application to not hang yet still display a GtkStatusIcon-powered tray icon would be to run with `dbus-user none`, and writing the following in `~/.config/JetBrains/Toolbox/toolboxrc` to start a dummy non-proxying D-Bus server in the sandbox:

eval $(dbus-launch)
export DBUS_SESSION_BUS_ADDRESS
./$app "$@"
kill $DBUS_SESSION_BUS_PID
exit

@mid-kid Thanks for the details.

(Re-closing as "not planned" since the bug is not in firejail)

"Invalid client serial" when used with xdg-dbus-proxy
Hi, I was trying to use Jetbrains Toolbox with xdg-dbus-proxy, and I encountered an issue with your library.

Specifically, if xdg-dbus-proxy's --filter option is being used, the app will hang on startup for about 5-10 minutes before continuing. Meanwhile, xdg-dbus-proxy spits out the following message:

** (process:10596): WARNING **: 21:28:17.026: Invalid client serial

## steps to reproduce

Download the Jetbrains Toolbox app, then run the following:

xdg-dbus-proxy "$DBUS_SESSION_BUS_ADDRESS" "$HOME/dbus" --filter --log &
DBUS_SESSION_BUS_ADDRESS="unix:path=$HOME/dbus" ./jetbrains-toolbox

Full xdg-dbus-proxy log:

C1: -> org.freedesktop.DBus call org.freedesktop.DBus.Hello at /org/freedesktop/DBus
B1: all signal com.canonical.Unity.LauncherEntry.Update at /

** (process:15395): WARNING **: 12:49:58.552: Invalid client serial

## temporary workaround
run jetbrains-toolbox like this instead:

./jetbrains-toolbox --appimage-extract && cd squashfs-root
rm lib/dbus-java-transport-native-unixsocket-4.0.0.jar
APPDIR="$PWD" APPIMAGE="$HOME/.local/share/JetBrains/Toolbox/bin/jetbrains-toolbox" DBUS_SESSION_BUS_ADDRESS="unix:path=$HOME/dbus" ./AppRun

This causes the app to never connect to dbus, bypassing the hang.

## cross-linked issues
 URL
 URL

Sorry I don't get it...
What are you trying to do and where is it related to dbus-java?
What is xdg-dbus-proxy and why should I use it?
Which version of dbus-java is involved here?
Are you using it as client or is it running the embedded dbus?

xdg-dbus-proxy is a utility used for separating apps that use dbus from the rest of the system, giving them only access to the specific services that they need. It creates its own dbus server and connects to the system's dbus server to proxy and filter calls. This is incredibly useful for any kind of sandboxing technology, such as flatpak and firejail.

I am trying to get an app that uses dbus-java to run inside a sandbox whilst using xdg-dbus-proxy to restrict its access from most dbus services. My current trail of debugging has led me towards this library, as removing its transport jar changes the result of the execution, causing it to not connect to dbus at all, instead of hanging.

The version of dbus-java involved, according to the jar files in jetbrains-toolbox, is 4.0.0.

Is there any sort of log produced by dbus-java? Usually dbus-java uses Slf4j with any slf4j compliant logger to log what's going on. Especially turning on TRACE-logging would definitely help to find out what is going on.

Are you sure you are xdg-dbus-proxy the correct way?
Just using --filter will cause xdg-dbus-proxy to block all communication except for the initial handshaking to DBus.
See here.

When using xdg-dbus-proxy and a patched version of jetbrains-toolbox (with logging enabled) I can see that dbus-java connects to the Bus for the handshaking stuff, but as soon as jetbrains-toolbox want to listen to signals of com.canonical.Unity.LauncherEntry the signal is not being answered and the socket is closed.
For me it looks like if there are missing a lot of options for xdg-dbus-proxy to tell the proxy to allow talking to com.canonical.Unity.LauncherEntry and may others as well.

I wasn't able to tell which dbus services it was trying to access, but yeah I reckon it needs a few whitelists to work. Still, I don't have a com.canonical.Unity.LauncherEntry on my system, so it should have the same behavior here? And if DBUS_SESSION_BUS_ADDRESS is set to some bogus value like "foo", it gracefully and silently fails as well..

Thanks a lot for digging into this issue! Glad to know the mechanics of all of this now. It makes sense, as xdg-dbus-proxy works in a way many applications don't expect, but I guess I was confused since I've never seen similar behavior in other programs using dbus. Many thanks for implementing a workaround anyway, I'll be patiently waiting for that to trickle downstream :)

And yeah, the "Invalid client serial" message comes from xdg-dbus-proxy itself: URL
I wrongly assumed that would be the issue since I didn't see this message pop up during correct operation. It might not be related at all.


Question: Unable to build due to "clang++ exited with error code 1" and empty file?

It's a known issue, and there's a proposal to improve this scenario: URL (which also explains the reason why it behaves like it does today).

Unable to build due to "clang++ exited with error code 1" and empty file
### Apple platform

iOS

### Framework version

net9.0-*

### Affected platform version

VS 2022 17.13.1

### Description

1>C:\Program Files\dotnet\packs\Microsoft.iOS.Sdk.net9.0_18.2\18.2.9180\targets\Xamarin.Shared.Sdk.targets(1665,3): error : clang++ exited with code 1:
1>C:\Program Files\dotnet\packs\Microsoft.iOS.Sdk.net9.0_18.2\18.2.9180\targets\Xamarin.Shared.Sdk.targets(1665,3): error : ld: file is empty in '/Users/**/Library/Caches/Xamarin/mtbs/builds/**/e714b99856d2d8abeed05f03af7a2bd21b67f244c175af00f39b5193cbd8bd41/C:/Program Files/dotnet/packs/Microsoft.NETCore.App.Runtime.Mono.ios-arm64/9.0.2/runtimes/ios-arm64/native/libmono-component-marshal-ilgen-static.a'
1>C:\Program Files\dotnet\packs\Microsoft.iOS.Sdk.net9.0_18.2\18.2.9180\targets\Xamarin.Shared.Sdk.targets(1665,3): error : clang++: error: linker command failed with exit code 1 (use -v to see invocation)

I checked that path on the Mac and it is indeed a 0 byte file. All the other files in that directory are non-zero in size.

The corresponding file on my Windows machine is 199KB.

I scp'd that file from my Windows machine to the Mac and that allowed the build process to get past that error and it seems to fully build, though it can't deploy due to an issue with the iOS version.

### Steps to Reproduce

I don't know. I have had to do so many things to get to this point with a brand new Mac and an app that has never been built for iOS before.

### Did you find any workaround?

Yes, I copied the file from my Windows machine to the Mac, overwriting the 0-byte file.

### Build logs

_No response_

Actually, that deploy issue also seems to be a bug. When I added `MinimumOSVersion` to my Info.plist file, I get the following error:

But I have the following in my project file:

xml
 13.0

I was able to get rid of that error and deploy and run by changing the above to:

xml
 13.0

Using the entire target framework string instead of the platform identifier.

Then I added a msbuild target to output $(TargetPlatformIdentifier) and that outputs "MacCatalyst".

I'm not sure if these two issues are related.

I think I reported the main issue in the wrong place, so I'm closing this and moving the issue with $(TargetPlatformIdentifier) to #22238.

$(TargetPlatformIdentifier) is not being set correctly for iOS
### Apple platform

iOS

### Framework version

net9.0-*

### Affected platform version

VS 2022 17.13.1

### Description

I split this from an issue I created earlier: #22235, but I am going to close that because I think I added it in the wrong place and need to report that through the VS Feedback feature. I'm also not sure I am adding this issue in the right place.

I have a project configured with the following:

xml
 13.0
 13.1
 23.0

The last line, for Android, works. The MacCatalyst line also appears to work, though I haven't tested it all the way to deploying.

The first line does not work when deploying to an iOS with an OS version below 18.2. The condition is not met and so the `SupportedOSPlatformVersion` defaults to 18.2.

Adding a `Target` to the project file to output the value of $(TargetPlatformIdentifier) has some strange behavior.

When doing a solution rebuild it will output a 'MacCatalyst' (this is why I think the above line for MacCatalyst works) and 'Android'. It will not output 'iOS'.

However, if I do a build by running the app on a remote iOS device, it only outputs 'iOS', as expected. But after that the deploy fails due to the device not having the minimum required OS version of 18.2. It looks like mtouch is what outputs the error. The error is long and formatted all on one line, but it contains the following messages:

The following configurations do work:

xml
 13.0
 13.1
 23.0

and

xml
 $([MSBuild]::GetTargetPlatformIdentifier('$(TargetFramework)'))

 13.0
 13.1
 23.0

For some additional information, if the minimum OS version is set in Info.plist to "13.0" then an error (or warning? I don't remember) is output stating that the value conflicts with the value from the project of "18.2", indicating that it isn't getting picked up and is being defaulted to 18.2.

This part might be subjective, but it seems strange to default the minimum OS version to the maximum OS version. I would think the default OS version should be the minimum OS version supported by the framework. To that point, the configuration from the MAUI template sets `SupportedOSPlatformVersion` to "11.0". It seems like that should be the default for `SupportedOSPlatformVersion` isn't specified at all.

### Steps to Reproduce

Create a MAUI 9.0 project and add the above lines to the project file. Run the project on a device running a version lower than iOS 18.2. It will build, but not deploy.

### Did you find any workaround?

Yes, the last two configurations described above.

### Build logs

_No response_

This is a bit counterintuitive, but the problem is `TargetPlatformIdentifier` isn't set at this point in the build process - so the question becomes "why is it working for Mac Catalyst and Android?"

I was able to test both Android and Mac Catalyst, and it doesn't work on either platform. This is easy enough to see with a non-sensical values:

xml
 1.0
 1.0

and the build succeeds just fine, for both Android and Mac Catalyst.

changing to:

xml
 1.0
 1.0

raises a build error on both platforms.

I'm not sure why it seems to work for you, but it might just be that the default value kicks in and you don't see any problems with it.

We've had different opinions on this internally, and there doesn't seem to be an obvious default logic that works across all platforms. In fact it seem to depend on how old the minimum OS version actually is - would you expect the default to be the minimum iOS version if the minimum was something ridiculous, such as iOS 2.0? This isn't entirely hypothetical, apparently this happens when building for Windows, the SDK supports some rather old versions of Windows.

In any case, I just wanted to acknowledge that your point is something we're aware of, and we may end up changing in the future.

I see, thanks for the explanation. Now that you mention it, the others not working makes sense because I remember before this that changing the value wasn't getting rid of some warnings to version guard some code that I thought it should. Now I know why. I guess that should have clued me in.

But shouldn't it be set that early in the build process? Otherwise, what use is it? It doesn't have a _ prefix to indicate it is internal. So now my question is why does that exist if it can't be used?

As for the supported versions, I'll just say that the issue might not be that it doesn't support the minimum by default, but that it only supports exactly the last version by default, so a solution that just handled that differently would probably be better (for example, defaulting to last version - 1 or something like that).


Question: Run DLL error after start?

@mr-brunes
@harkinhails

Recently there were major upgrades in the Python framework that may help. Would you please test one of these options?

1. BleachBit 4.9.0 alpha is based on Python 3.10 and is digitally signed
2. BleachBit 4.9.1 CI build 2775 , based in Python 3.11, is newer but is not digitally signed.

Both have known issues, so may not be ideal for everyday use. However, testing would tell us a lot about whether the Python upgrade fixed the issue.

A workaround is install BleachBit in a folder without a space such as
* `c:\bleachbit`
* `d:\bleachbit
* `d:\program_files_x86\bleachbit`

Please let me know what you find out.

Run DLL error after start
**Describe the bug**
After start of bleachbit I see a popup with an error message "the module was not found"

**To Reproduce**
Steps to reproduce the behavior:
1. Install Bleachbit
2. Start Bleachbit
3. See error

The error happens every time.

**Expected behavior**
No error popup.

**Screenshots**
Image

**Desktop (please complete the following information):**
 - OS: Windows 11
 - BleachBit version 4.6.2

@chrisonline This looks like issue 1685 "The specified module could not be found". Recently there were major upgrades in the Python that may help. Would you please test one of these options?

1. BleachBit 4.9.0 alpha is based on Python 3.10 and is digitally signed
2. BleachBit 4.9.1 CI build 2775 , based in Python 3.11, is newer but is not digitally signed.

Both have known issues, so may not be ideal for everyday use. However, testing would tell us a lot about whether the Python upgrade fixed the issue.

A third option is to install BleachBit in a folder without a space such as
* `c:\bleachbit`
* `d:\bleachbit
* `d:\program_files_x86\bleachbit`

Please let me know what you find out.

@az0 With **BleachBit 4.9.0 alpha** I don't get this error anymore. So I will wait until 4.9.0 is released. Thanks.

"The specified module could not be found." - It's back from the dead.
Same issue fixed here, introduced back again in 4.6.0.

I am wondering whether it was in fact fixed for 4.4.0 as stated in the related ticket, so do BleachBit 4.4.0 and 4.4.2 have this issue on your system? See old downloads.

4.6.2 still has this issue "There was a problem starting D:\Program ... The specified module could not be found."


Question: New styles for footer actions?

@Nazi-pikachu no as you can see in my description above this is blocked by two other issues.

New styles for footer actions
Here are the styles of this specific component:

Image

And more generally of all dropdowns like this: URL

Ideally we’d only get to this after @wagtail/gsoc-ux-unification has implemented their changes to the JS side of this.

Note for footer actions – there is much more than just the dropdown, we also need to make sure the new styles work with the bulk chooser’s components, and with our dirty form check messages.

As part of this, we could also add this component to the user creation screen to help with #5906. Ideally #7366, #6591, #7650 will have also been fixed.

Sub-tasks:

- [x] Re-implement dropdown buttons with Tippy
- [x] Re-implement bulk actions + button styles according to new designs
- [x] Include styleguide improvements for warning buttons URL

Hi @thibaudcolas I was wondering if this is still open for anyone to work on?

Hi @ForJessicaSake, yep, I don’t think anyone else has worked on this.

Okay great, I’ll work on it can you please assign it to me 😊 thank you!

@ForJessicaSake we do not assign issues (outside of core team) so feel free to just start on this.

Hi @ForJessicaSake, can you let us know where you got to with this? Please let us know if you have any questions.

@Seremba feel free to give this a go as we have not heard back from @ForJessicaSake for some time.

Hi @thibaudcolas @lb- would you mind helping me locate the file for these changes?

@Seremba have a look at this PR - it touches the same area of the UI.

 URL

However, it's good to learn how to find these things yourself. The best way is to look at the CSS classes in the DOM and search through the code in your editor.

I’ve done quite a bit of the work on this in #11456 – the footer actions code is much simpler now, so should be easier for us to complete this ticket. There were a few snags:

- The proposed widths for the sticky actions would mean a lot more wrapping of text to two lines for languages longer than english (so I stopped at 310px rather than the 250px in the designs)
- The designs in Ben’s original screenshot here have too low contrast between the footer actions background and the page background in my opinion. Not sure this is an actual issue but I found it hard on the eye to not know exactly what part of the page was sticky and what was within. Something I’d like to discuss at least.

And more generally:

- Those styles need to work for both footer actions on "forms", and bulk actions. Currently we only have designs for very simple "form" footer actions, while the bulk actions are more complex visually
- We need to make sure the footer actions also support having text within, the "unsaved page data" warning and also meta text for bulk actions.

---

Another snag was that footer actions markup currently is copy-pasted all over the place, with lots of inconsistencies. So as part of completing this I think we need to refactor the code to have an implementation that’s easier to reuse. Particularly in the context of #9869.

Assigning back to @benenright for a review of how we can tweak the designs to complete this, based on my notes above. Here are specific problem scenarios that might warrant design changes.

## Footer actions with warning message

footer actions with warning text

## Bulk actions

bulk actions

## Bulk actions with "Select all" link

bulk actions select all

I think that we should keep the background indigo, as I don't think there is much value in trying to make a light background work behind these actions, and the potential usability issues outway the original intention...

The original intention was to reduce the visual weight of the form footer actions - being something that sticks on the browser bottom edge, it felt like this was worth trying. But in the context of the bulk actions, that only appear once triggered by selecting a checkbox, the dark background helps users to notice it, and if we move to a lower contrast background, it risks them not being noticed (some usability testing we did a while back showed that some users expected these bulk actions to appear at the top of the listing (as per Gmail for eg), but since the indigo bg was high contrast with the page bg, people did notice it at bottom).

I do think it would be worth revisiting how we ensure good contrast for this type of panel specifically in dark mode (as the low contrast means we likely have that usability issue where people don't notice it appear):

image

@benenright I forgot one other variation on footer actions, which is used when viewing a past version of a page:

footer actions past revision

This is testable on this past revision demo page. Could you give us design guidance for that as well?

---

re bulk actions, anecdotally we received related feedback from low-vision users (though it’s low on details so I’m following up):

Hi

In this view, I don't know the history as to why the button is orange, but to me it feels like the top banner is doing enough to alert people this is a previous version of this page. I feel like for that reason we _could_ just revert the buttons to be teal here...

However, maybe it's likely that people who have become used to seeing the buttons orange will get confused if they go back to teal). So for that reason it's prob safer just to address the fact that the white text on orange is not high enough contrast... and we should change it to use the same text colour as in the alert banner.

Image

----

While we're here. could we fix this UI glitch where the button loses it's top left and top right border radius after you open then close the options:

Before:
Image

After:
Image

👍 sounds good to me that we’d have the button consistently teal. As you say the banner is already doing a good job warning users, and the button text is different as well. Also agree re keeping the indigo background in the light theme.

Will leave it with @laymonage to figure out where to place the "Delete" buttons we occasionally have in forms, that could either go in a "footer actions" dropdown, or in a "slim header" falafel actions menu 👀

Implement sticky form actions across all forms
### Is your proposal related to a problem?

See #9760 for further context. Blocked by #9030, #7366. On some of the forms in the CMS, we have the "Submit" and other action buttons at the end of the form, while on others we use a "sticky footer actions" component. For consistency, we’d like to switch all forms to "sticky actions" wherever possible.

### Describe the solution you'd like

- The Save/Submit button of a form should be sticky at the bottom of the screen.
- Any secondary action buttons ("Delete") should live within the dropdown

### Describe alternatives you've considered

See UX research in #9760.

Hi @thibaudcolas i would like to work on this issue, is it up for grab?


Question: There is a probability of crashing when using Warhammer right-click to attack creatures?

Alright, I've looked into this again. Not sure where my head was at before, but I've reverted my previous changes, applied the suggested changes, and released a new version.
Thanks again @XFactHD for your input!

There is a probability of crashing when using Warhammer right-click to attack creatures
version 1.20.1
forge 47.3.1
weaponmod-forge-1.20.1-1.21.3
architectury-9.2.14-forge
cloth-config-11.1.118-forge
There is a probability of crashing when using Warhammer right-click to attack creatures.
I really like this mod.
Also, my English is not good, so I used mechanical translation.

I tried to reproduce this bug again and found that it only crashes when the focus is on the creature.
This is the log of the game crash. I hope it can help lock in the bug.
minecraft-exported-logs-2024-07-01T10-47-35.log

I am trying to reproduce this currently, but I am unable to do so yet.
Your logs are cut-off sadly. Could you try posting the full file?
Also, the file `G:\game\game2\我的世界\.minecraft\versions\1.20.1ff\crash-reports\crash-2024-07-01_10.47.10-client.txt` might help

Sadly, this is still cut off - I need everything that is above ` at cpw.mods.modlauncher.LaunchServiceHandlerDecorator.launch(LaunchServiceHandlerDecorator.java:30) ~[modlauncher-10.0.9.jar:?] {}`

Your current crash log does not show something specific to BWM: Legacy yet

Sorry, I don't quite understand this. The previous logs were automatically exported by an unofficial launcher. This is the log I found in the logs folder
latest.log

Now I can see the full crash log, thanks :)

This is the same issue as URL so the fix should be rather analogous

IllegalStateException: Accessing LegacyRandomSource from multiple threads
#### Issue type:

- :bug: Bug

____

#### Short description:

Game crashes due to blood_splash particle colliding with blood_stain block.

#### Exact Error:
The game crashed whilst accessing legacyrandomsource from multiple threads
Error: java.lang.IllegalStateException: Accessing LegacyRandomSource from multiple threads

Same as #954 and #949 in the ATM8 modpack.

#### Steps to reproduce the problem:

1. Load the game, play for a while
2. Mobs receive damage by falling or in a mob farm
3. Game crashes due to collision of blood particle and blood stain

#### Expected behaviour:

The game should not be crashing randomly due to blood stain block and blood splash particle interaction.
____

#### Versions:

- This mod: EvilCraft 1.2.12
- Minecraft: 1.19.
- Forge: 43.1.55
- ATM8 Modpack: 1.0.7

#### Log file:

crash-2022-12-26_13.19.15-server.txt

EvilCraft isn't using `LegacyRandomSource` anywhere anymore since #954 as it isn't thread-safe.

However, Minecraft itself _is_ using `LegacyRandomSource` for spawning particles. So if another mod is using `LegacyRandomSource` incorrectly, and Minecraft spawn a particle, a delayed crash will occur. So the other mod that is using `LegacyRandomSource` incorrectly is at fault here.
Other particle-generating mods will likely result in a similar crash due to this, even though they are not at fault.

I recommend reporting to your modpack author, and asking them to add/remove mods until they find the offending mod, and reporting it to them. There's not much I can do on my end unfortunately.

Since this issue is still happening in the latest version of AllTheMods 8 (1.0.9), further research into this was done and it was found that EvilCraft is indeed at fault here despite the previous fix. The previously deployed fix doesn't achieve anything regarding this issue as the crash doesn't happen directly in any of the code touched by the fix since that code did use the correct `RandomSource` from the `Level` given by the caller. The actual crash always happens in Minecraft code when the `LevelRenderer` tries to apply some particle amount randomisation because it tries to use the client level's random from the server thread where your code unintentionally calls it from.

The underlying issue causing this crash is that the spawning of the blood splash particles ( URL does not check for the logical client side, it only checks for the physical client side, both explicitely via the helper call (checks `FMLEnvironment.dist == Dist.CLIENT`, which is purely the physical side) and implicitely via the `@OnlyIn(Dist.CLIENT)` annotation. This leads to the particle spawning code being called from the server thread in singleplayer as is visible in both the stacktrace linked by the OP and this log snippet where the logging of the `LegacyRandomSource` was extended via Mixins for debugging purposes. The proper fix for this is to replace the helper call with a call to `Level#isClientSide()` in the method linked above, which would prevent the code from reaching across logical sides and crashing in the `LevelRenderer`'s random shenanigans.

It's also worth noting that the previous fix is potentially unsafe since it deploys a global and completely unguarded `SingleThreadedRandomSource`, which may produce weird results when accessed from multiple threads, which is the reason why the `LegacyRandomSource` crashes in these concurrent access scenarios.

Thanks for the findings @XFactHD! Will have a look at this soon.


Question: [PT Run] [Time and Date plugin] Allow custom format string?

@Picazsoo
Great request. Feel free to open the PR. Your contribution is welcome. If you need any help, you can ask me.

I personally prefer `YYYY-MM-DD_HH-MM-SS` in my file names. Not sure what's better or if we need both variations/a setting.

[PT Run] [Time and Date plugin] Allow custom format string
### Description of the new feature / enhancement

The PR #16662 implements a new plugin for date and time values.

It might be interesting to support custom format string as user input like the wox plugin "DateTimeFormatter".

### Scenario when this would be used?

If the user needs an unsupported format.

### Supporting information

 URL

---

⚠ The work on this enhancement can't begin before the plugin code is merged into main branch!!

Hi @htcfreek , I think this is a really good idea. Even better would be to _also_ be able to manually add custom format string in the Date and Time plugin settings. So the user could specify their own custom format and always have it appear as the very last result (hence possible to navigate directly to it by pressing up).

The reason this functionality would be pretty useful is because currently none of the include date & time formats can be directly pasted into a Windows file / directory name, since they include either forward slashes or colons. I created an issue #24911 for a proposal for a simple addition of one more date time format, but more robust solution with custom persistable format would be preferable I think.

I have searched for this and can't find it, so apologies if it's a duplicate suggestion.

I spend a lot of time with one foot in the USA and the other in Great Britain or Europe, and that means dates being misinterpreted because of the **mm/dd/yy** vs **dd/mm/yy** thing we are all familiar with.

This multicultural issue has led me in my daily life to use the format you see on passports and some American driver licenses:
DD-MMM-YYYY, for example, today is: **3 MAR 2025**

It is impossible to misinterpret a date written in this format, and I've taken to formatting my dates this way in Excel and in correspondence. I mostly use a short "long form" in my correspondence when adding days of the week: **Mon 3 Mar 2025**

**The cultural issue, which I suspect many people have, particularly Americans working overseas:**
Having worked on loads of US based accounting documents in my career, my habit is to enter dates on the numerical keypad in US format. So while I **_write_** today's date as Mon 3-Mar-2025, I **_enter_** today's date in Excel as 3/3/25.

So far, so good - Windows can do this, and Excel can do this, as it stores dates as numbers and we have loads of formatting options including just this issue. _But that is only true if my Windows settings are set to USA, like this:_

But I really struggle with that - I **want** my Windows system date to look like this:

_**Actually, I would prefer the system date to read "**Mon 3 Mar 2025**" but I can't find a way to produce this "short" longform**_

**Problem #1**
When you dig into the "English (world) settings, there is no short or long form option that abbreviates the days of the week to Mon, Tue, Wed, Thu, Fri, Sat and Sun. The longform system date that includes the day of the week uses too much real estate.

It would be terrific if we could add an abbreviated day of week to the short date option.

**Problem #2**
When you have set the system date parameters to the "world" settings, your excel spreadsheets no longer provide a formatting option which permits data entry in dd/mm/yy format. Your entire Excel world is now expecting data entry in mm/dd/yy format and its derivatives.

I cannot for the life of me sort out how use the international system date format I prefer in combination with Excel accepting date entry in standard US format of mm/dd/yy with the option of displaying those dates in the "International Passport" format.

Perhaps I'm missing something despite having played around with Excel date and custom format options for ages.

Is this something that could be done? Or have I completely missed the standard options available to me?

Thanks

Well you bring me to an following idea that could be the best solution. We add a multilline text setting that allows the following:

MyFormat1=YYYY-DD-MM
MyFormat2=DD-MMM-YYYY
....

Thank you.

Since my programming days pretty much ended with COBOL, FORTRAN and dBase
III with a touch of SQL in the 1980s, I'll not annoy you with code related
suggestions, but perhaps this clarification helps:

1. Excel appears to derive what it expects as date *entry* formats from the
system date setting.
2. Excel offers different date *formatting *options based on the formatting
of the system date.
3. A logical fork which allows *formatting* the displayed system date *to
be different *from the instructions Excel receives might prove useful in
getting the end result I'm suggesting.
4. Adding more examples to include day of week to your multiline text:

MyFormat3=DOW-DD-MMM-YYYY
MyFormat4=DOW_DD_MMM_YYYY

So put another way, what I'm suggesting is a *separation* between the
system date format as shown on the Windows taskbar and the system date
format as read by Excel (and likely other Microsoft products). Either
that, or modify the date options code in Excel that permits the same level
of customization.

To follow any of this, one must understand something that I find very odd
and might be missed at first reading, so I'll repeat it:

While Excel stores dates as numbers and can format their display in many
different ways, Excel does not appear to offer options as to how those
dates are entered. So if the system date is set to "World", Excel expects
you to enter dates in dd/mm/yy sequence. I propose a bifurcated approach,
allowing display of system date in World format and data entry of dates in
US format.

I hope that is clear! And if you got all that the first time, I apologize
for repeating it. Thank you again.

On Mon, Mar 3, 2025 at 3:55 PM Heiko ***@***.***> wrote:

--
Mark Merritt
***@***.***
505.270.5522

Filename-compatible datetime format in PowerToys Run Time and Date plugin
### Description of the new feature / enhancement

The _Time and Date_ plugin functionality in _PowerToys Run_ is very handy for quick obtaining of current date / date and time. However none of the existing options are compatible with Windows filenames constraints. That means that it is not possible to quickly obtain a date-time in format, that could be easily pasted into a filename/directory name when e.g. downloading a file from the internet or simply saving or renaming any file or creating/renaming a directory.

I propose a solution by adding additional filename-compatible version of the timestamp.
The date and time formatting should:
- be easily human-readable
- allow for good file sorting by name (year -> month -> day -> hour -> minute -> second)
- omit any characters forbidden in Windows file names (e.g. forward slashes and colons)

One such possibility would be `yyyy_MM_dd-hh_mm_ss`. However this is not set in stone and I am definitely open to suggestions.

I am happy to provide a PR with the changes if there is agreement that this would be a sensible addition to the project.

### Scenario when this would be used?

This would be used when needing to obtain human-readable current date and time that can be embedded in directory/file name. Currently no such format is available.

### Supporting information

_No response_

Hi @stefansjfw Stefan,
I disagree that this is a dupe of the issue #16934. In that issue the referenced Wox plugin's behavior is that a user can type the requested format into the prompt and have the current date formatted as such. That is obviously cumbersome for day to day use, since one would have to retype the required format string on every single invocation. What I propose is a permanent addition of one additional format in the results that would provide easily pastable filename-compatible format.

But I can continue the conversation in the other issue I guess if it is inappropriate here.

Best regards,
Jachym

@Picazsoo Ok, I was considering linked issue as a more generic solution. Reopening the issue.

Hi @htcfreek, I have no problem with your proposed format. I am not sure that including both options would be a good idea, because these two are not the two only valid options anyway and even including just two variations would already bloat up the results list. So if you agree I would stick to your format for now.

I have more ambitious implementation working that allows enabling & customizing the date & time format string in the settings, but I would prefer to only do that after this simple implementation as the changes are much larger and I expect some understandable push-back on that one.

@Picazsoo
No need to stress you. Please update the unit test project too.

I had to change from `yyyy-MM-dd_hh-mm-ss` to `yyyy-MM-dd_HH-mm-ss`. Otherwise the hourly value in the timestamp would be terribly ambiguous - therefore useless.

Fixed in 0.69 which was released in the start of April 2023.


Question: InputImageRotation issues: Vertical images are not working, etc...

Closing this issue.

Fix: google_mlkit_text_recognition instead of `google_ml_kit`.

More details here: URL

Moving `InputImage` and `InputImageRotation` issues to: URL

InputImageRotation issues: Vertical images are not working, etc...
In Flutter I am trying to detect text from images in horizontal orientation is working perfectly but Vertical orientation text is not detected.
I am using the google_ml_kit plugin.
Anyone has a solution for this or another way to achieve this functionality.

Here you can check it is working with native android :- URL

Same Problem here.

I am using the example app provided by the package author.
It works fine in portrait mode but if I rotate the device all the API's stop working or acting malfunction.

To be specific, I am using Samsung Galaxy Tab A7 SM-T505 right now and it is not working OK.

To test if there is a problem in imageOrientation I just added a log print over this function inside _processCameraImage:

...
final imageRotation = InputImageRotationValue.fromRawValue(camera.sensorOrientation) ?? InputImageRotation.rotation0deg;
 print('orientation: ${imageRotation.rawValue}');
...

and find that it is always 90 even if I rotate the device it will always detect my device in landscape mode.
I don't know if I am testing and point in correct direction but I think it is something related to the cameraOrientation which brings this bug.

I am using the last version of camera and recent update of ml_kit.

google_ml_kit: ^0.10.0
 camera: ^0.9.4+20

This issue has to do with the orientation of your device/camera/inputImage. The example app is not production code and only works on portrait mode. Feel free to modify the example app to support landscape, but you need to edit this method: URL

I have not played that much with the orientation and metadata, but you need to create the correct instance of `InputImagePlaneMetadata` and `InputImageData` to pass to `InputImage.fromBytes`.

Once you figure it out come back and share the fix here or create a PR, contributions are always welcome.

Another comment in another discussion that might help you to resolve your issue is here: URL

As you mentioned earlier, I believe it is something related to camera package which cannot find the correct aspect ratio after orientation change.
To be exact, the imageProcess is working on something different than what you have in cameraPreview because, if you move the camera at the edge, where the barcode is merely outside the cameraPreview, you can see that it can detect the barcode correctly. SO, I think the image which is processing has different data than what we have in the cameraPreview.

I hope I could investigate further to see if I can match the zoom level of Landscape and Portrait in cameraPreview to see if the issue is resolved or not.

Any suggestion is highly appreciated.

From my view the current culprit is this widget from camera_view.dart:

` Widget _liveFeedBody() {
 if (_controller?.value.isInitialized == false) {
 return Container();
 }

 final size = MediaQuery.of(context).size;
 // calculate scale depending on screen and camera ratios
 // this is actually size.aspectRatio / (1 / camera.aspectRatio)
 // because camera preview size is received as landscape
 // but we're calculating for portrait orientation
 var scale = size.aspectRatio * _controller.value.aspectRatio;

 // to prevent scaling down, invert the value
 if (scale [

 Transform.scale(
 scale: scale,
 child: Center(
 child: _changingCameraLens
 ? Center(
 child: const Text('Changing camera lens'),
 )
 : CameraPreview(_controller),
 ),
 ),
 if (widget.customPaint != null) widget.customPaint,
 Positioned(
 bottom: 100,
 left: 50,
 right: 50,
 child: Slider(
 value: zoomLevel,
 min: minZoomLevel,
 max: maxZoomLevel,
 onChanged: (newSliderValue) {
 setState(() {
 zoomLevel = newSliderValue;
 _controller.setZoomLevel(zoomLevel);
 });
 },
 divisions: (maxZoomLevel - 1).toInt() < 1 ? null : (maxZoomLevel - 1).toInt(),
 ),
 )
 ],
 ),
 );
 }
`

The landscape zoomed picture problem rooted from this part of the code:

 Transform.scale(
 scale: scale,
 child: Center(
 child: _changingCameraLens
 ? Center(
 child: const Text('Changing camera lens'),
 )
 : CameraPreview(_controller),
 ),
 ),

If you change the `scale: scale` to `scale:1` it will fix the landscape orientation zoom problem but with some drawbacks.
The main cons of doing this is the aspect ratio of the camera preview, which looks to be not the same as the main scaffold's body.
After all, I think passing 1 to `scale` may solve the problem for now.

TextRecognizer works ok on Android but the result letters are all corrupted on iOS
Hi,

I got recommended to use google_mlkit_text_recognition instead of google_ml_kit as followes
 URL

However, it still works ok on Android but not on iOS using google_mlkit_text_recognition as using google_ml_kit

recognizedText.text look ok on Android but the letters in it are all corrupted on iOS
For now, I need only English TextRecognizer

 import 'package:google_mlkit_text_recognition/google_mlkit_text_recognition.dart';

 textRecognizer = TextRecognizer(script: TextRecognitionScript.latin);
 final File? imageFile = await getImage(ImageSource.gallery);

 if(imageFile != null) {
 final inputImage = InputImage.fromFile(imageFile);
 final RecognizedText recognizedText = await textRecognizer.processImage(inputImage);

 String text = recognizedText.text;
 print('${this.className} extractText : ' + text);
 }

google_mlkit_text_recognition 0.0.1

My development environment is as followes
iOS 14.7.1
Xcode 13.3

Please help me to solve this issue
Thanks in advance

@kodol73 : could you share an image of the text you are trying to recognize? I have been trying text recognizer and it works for me.

Here you have some screenshoot of the example app in iOS and it is working.

IMG_0050

IMG_0051

i have a same problem, it detect wrong text when run on IOS flutter

It seems that some images of text are recognized correctly, but some other images of text are corrupted
I just tried with example app, but has the same result

The result of recognizing below image is corrupted
Please check with the below image

IMG_1391

@kodol73 : I tried your image with the example app and so far it does not look corrupted to me. It looks pretty much good to me.

IMG_0052

You tried it on iPhone, right?
Can you please let me know your phone model and iOS version?

I tried with both an iPhone 11 and iPhone 6S, both with iOS 15 and I am using Xcode 13.3

it's weird, I just upgraded iOS to 15.4.1, but the same result
It generated corrupted text like the following with the image

T
T
ooo
5

Just in case, I test with google_ml_kit 0.8.0 because example app uses google_ml_kit

Please help me to solve this issue

I had this same issue, but resolved it on iOS.

Still testing android but I am passing the File from camera plugin cameraController.takePicture() as the InputImage.fromFile(imageFile) and looking at text it had almost no relevant data, and any data it did have was like:
A
ooD
TM

After removing the google_ml_kit plugin, and replacing it with only the google_mlkit_text_recognition: ^0.2.0 it works fine. No idea why to be honest, but specifying just text recognition instead of the umbrella plugin resolved. Would like to hear some details on this if there are any.

Thanks @codyfilatov for the suggestion.

Do you still have issues with Android? both platforms are working okay?

@kodol73 : give it a try

Both platforms are working okay now.
Although I did not have this issue with Android to begin with.

The only issue I am having now on iOS is the orientation of my image when processing, it thinks it is in landscape when it is in portrait, so the bounding boxes are messed up, does not happen on Android. My image does not have any exif orientation data. I would be nice to have if no orientation data exists to predict based on the height/width, looks like it defaults to Landscape if no data exists. But that's another issue I can figure out.

Hope OP resolves issue with the dependency change.
Thanks

@codyfilatov: once you figure the orientation out, share here or create another issue if necessary. I have seen issues with different orientations as well.

I just tested the newly released version of google_mlkit_text_recognition, 0.2.0, nothing else changed
The issue doesn't appear any more

Thanks

Sharing here just as a note.
The orientation issue was caused by no EXIF orientation data on the image in iOS taken from the camera plugin takePicture function. Since the image file had no EXIF orientation data I assume the text regognizer (and barcode scanner) was returning odd (not incorrect) values.

The values did not translate as expected as when I set a Container DecorationImage as the same file the image is displayed correctly as taken, but the values were printed mirrored and counter clockwise consistently.

After using the image package to set the EXIF orientation data to 1 and rewriting the bytes to a new file, then passing that file to MLkit the boundingBox values were as expected.

XFile imageCaptureXFile = await this.cameraController!.takePicture();
File imageFile = File(imageCaptureXFile.path);
this.imageFile = imageFile;
img.Image? image = img.decodeImage(await imageFile.readAsBytes());
if(image == null){
 Util.showToast('Image was not found.');
 this.isConfirming = false;
 this.imageFile = null;
 setState(() {});
 return;
}
if(!image.exif.hasOrientation){
 print('Baking orientation into image...');
 image.exif.orientation = 1;
 File orientatedImageFile = File((await getTemporaryDirectory()).path + Uuid().v4() + '.jpg');
 await orientatedImageFile.create();
 await orientatedImageFile.writeAsBytes(img.encodeJpg(image));
 await imageFile.delete();
 imageFile = orientatedImageFile;
 this.imageFile = orientatedImageFile;

 image = img.decodeImage(await orientatedImageFile.readAsBytes());
 if(image == null){
 setState(() {});
 return;
 }
}

This is what I use for my use case. Keep in mind I only want portrait.
I use the package:image/image.dart . Ignore the bad code, I'll clean it up later.

TextRecognizer textDetector = TextRecognizer(script: TextRecognitionScript.latin);
RecognizedText recognisedText = await textDetector.processImage(InputImage.fromFile(imageFile));


Question: SharePoint Online API: Access denied creating Personal Site when passing Signed In User Bearer Token?

Unfortunately the SharePoint Online support engineers are not always 100% up to date about the limitations of their own products... :-( In this case the issue lies with the fact that a requested permission through an Azure AD application grant is not being recognized by SharePoint online specifically when it comes to the UPS (User Profile Service Application). With -UseWebLogin we use a different type of authentication (cookiebased/legacy) where as with all the other authentication methods we use the currently by Microsoft promoted way of authentication : Azure AD OAuth. We switched over to Azure AD OAuth as the default as more and more API requests we need to make are located in the Microsoft Graph. And the MS Graph only supports Azure AD OAuth.

In order for the OAuth authentication flow to work one needs to grant permissions to an Azure AD App registration, either a multi-tenant one alike the PnP Management Shell, or your own registered one. Our tests show that while the correct permissions are in place, SPO is not allowing the user to perform the actions. We brought this up with the product manager in charge of those APIs back then but are still awaiting feedback.

SharePoint Online API: Access denied creating Personal Site when passing Signed In User Bearer Token
#### Category
- [ ] Question
- [ ] Typo
- [x] Bug
- [ ] Additional article idea

#### Expected or Desired Behavior

The below HTTP call should create a new personal site for the user

POST /_vti_bin/client.svc/ProcessQuery HTTP/1.1
Host: tenant-admin.sharepoint.com
Content-Type: application/xml
Authorization: Bearer

 user@tenant.onmicrosoft.com

#### Observed Behavior

Above request results in `200 OK` response but returns the below error, no personal site is created for the user.

[
 {
 "SchemaVersion": "15.0.0.0",
 "LibraryVersion": "16.0.19318.12002",
 "ErrorInfo": {
 "ErrorMessage": "Access denied. You do not have permission to perform this action or access this resource.",
 "ErrorValue": null,
 "TraceCorrelationId": "7c05089f-70d6-1000-af48-5aaa8564ad67",
 "ErrorCode": -2147024891,
 "ErrorTypeName": "System.UnauthorizedAccessException"
 },
 "TraceCorrelationId": "7c05089f-70d6-1000-af48-5aaa8564ad67"
 }
]

#### Steps to Reproduce

- Create Azure AD application
- Set the following delegate SharePoint Online API permissions
 - AllSites.FullControl
 - User.ReadWrite.All
- Grant Admin consent for delegated permissions
- Generate Bearer Token using OAuth 2.0 Authorization Code Flow
 - Provide Tenant Admin credentials
- Pass Bearer Token in Authorisation header of request

#### Related Issues

 URL
 URL
 URL

Any update on this?
Since ASC will be retired for SharePoint we'll have to use an Entra App Registration.
But that still doesn't seem to work...

Same challenge here, after discovering URL i was hopeful that using CreatePersonalSiteEnqueueBulk together with an entra id app registration and certificate will work, but sadly as of today it does not on our tenant and fails with the following error message:

Microsoft.SharePoint.Client.ServerUnauthorizedAccessException: 'Attempted to perform an unauthorized operation.'

Since many years we have been using SharePoint app only such as:
 URL

With the retirement of acs by 02.04.2026 this approach will no longer work, or am i misunderstanding the communication by Microsoft? What is the way forward here for the app only scenario, we really don't want to revert back to service accounts...

@andrewconnell @waldekmastykarz @garrytrinder thanks for your support!

[BUG] Set-PnPUserProfileProperty : Access denied. You do not have permission to perform
Please see also related discussion Can Get nut can't Set with pnp.powershell

### Expected behavior
Cmdlet `Set-PnPUserProfileProperty` allows to update of user-profile properties

### Actual behavior
Error is raised:

`
Set-PnPUserProfileProperty : Access denied. You do not have permission to perform this action or access this resource.
At line:1 char:1
+ Set-PnPUserProfileProperty -Account 'test@foobar.com ...
+ ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
 + CategoryInfo : WriteError: (:) [Set-PnPUserProfileProperty], ServerUnauthorizedAccessException
 + FullyQualifiedErrorId : EXCEPTION,PnP.PowerShell.Commands.UserProfiles.SetUserProfileProperty

`

### Steps to reproduce behavior
1. Connect SharePoint Online
2. Run following code: `Set-PnPUserProfileProperty -Account 'test@foobar.com' -PropertyName 'WorkPhone' -Value '0123456789'`

### What is the version of the Cmdlet module you are running?
`Manifest 1.3.0 PnP.PowerShell {Add-PnPAlert, Add-PnPApp, Add-PnPApplicationCustomizer, Add-PnPContentType...}`

### Which operating system/environment are you running PnP PowerShell on?
- [x] Windows
- [ ] Linux
- [ ] MacOS
- [ ] Azure Cloud Shell
- [ ] Azure Functions
- [ ] Other : please specify

How did you connect to SPO? i.e. which switch on the "_Connect-PnPOnline_"?

I have noticed a similar issue this morning, i.e. "**Access denied**." when using the "_-UseWebLogin_" or "_-Interactive_" arguments. However, when I connected via "**-Credentials**" on the "_Connect-PnPOnline_", it worked.

Perhaps the issue is with the Connect-PnPOnline cmdlet.

The issue is most likely connected to the registration of the PnP Management Shell application that is in place in your Azure AD. Run `Register-PnPManagementShellApplication` again, it will change the granted permissions: we added the userprofile readwrite right there like 2 weeks ago (it used to be only read access).

I am using the following approach to connect to SharePoint Online:

`
Connect-PnPOnline -Url ' URL -Interactive

`

@erwinvanhunen We registered the application yesterday and the permissions of this application seems ok:

image

I just tested and indeed it doesn't work with a bearer token (which is what we use by default). It seems that that is (not more?) supported. We'll investigate that. I noticed that if you use -UseWebLogin (which is cookie based auth) it does work.

I'll leave this issue open while we investigate.

My customer needs a fix for this! Here are a couple of possible clues, though I still get the same error behavior discussed in this issue.

Accessing SharePoint using an application context, also known as app-only says "User Profile CSOM write operations do not work with Azure AD application - read operations work. Both read and write operations work through SharePoint App-Only principal". Then Granting access using SharePoint App-Only gives instructions for setting that up. I was feeling pretty good when I found this, but then when I tested, still "Access denied. You do not have permission to perform this action or access this resource. Microsoft.SharePoint.Client.ServerUnauthorizedAccessException, ServerErrorCode : -2147024891, ServerErrorTypeName : System.UnauthorizedAccessException".

**Another possible clue** is that Granting access using SharePoint App-Only shows **-AppId** and **-AppSecret** parameters for Connect-PnPOnline, but the current version (1.3.0?) doesn't provide those parameters, only -ClientId and -ClientSecret.

_Is there a PnP core version I can test with that provides -AppId and -AppSecret?_

@SebasT87 what version of SharePointPnP are you using as a workaround?

@veronicageek, my apologies for seeming demanding, the exclamation point is a bit overused these days! I absolutely appreciate everyone's supporting each other, and I'll post something if I find an answer. I'm trying some things in this article now: URL

Can anyone share what **Connect-PnPOnline** parameter set and values to use to authenticate from an Azure Function App so the script can _write_ User Profile properties without requiring an interactive login?

I used **Register-PnPManagementShellAccess** successfully and it appears to be configured correctly. But I'm unsure how to tell Connect-PnPOnline to use the **PnP Management Shell** app that it installs, except with -Interactive which only works locally, not from an Azure Function App.

Also tried these approaches to no avail. All can read SP profile properties but none can write:
Connect-PnPOnline docs Example 6 (why doesn't this work if the app has SharePoint | User.ReadWrite.All | Application | Read and write user profiles?)

Granting access using SharePoint App-Only (may be deprecated, but moot because it doesn't work though it says it should for this specific scenario)

Many thanks

Finally found a way to authenticate to SharePoint online to _write_ profile properties without an interactive/user logon, that works from Azure function app. Unfortunately it uses ACS which is retired now but _explicitly still supported for SharePoint use_. So it's not my preferred approach, but as it actually works, it's better than all other approaches so far.

The key is to follow this article Granting access using SharePoint App-Only, but to add the scope /sharepoint/**social**/tenant. Update the Permission Request XML as follows, and it will permit writing user profile properties.

Still looking for a more "modern" approach that's not just supported as a waiver from a retired approach. Many thanks to all!

Hello, Is there an ETA on this Bug when it will be resolved ?

Like @veronicageek said upthread, all of this work is done by volunteers in their spare time. There usually isn't an ETA and some bugs don't get fixed.

To give you an update: this is not an issue with PnP PowerShell but has to do with how SharePoint Online handles authorization. We provided Microsoft with this feedback and are waiting for them to reply on it. We have no ETA when that will happen.

I tested it with `-WebLogin` and it indeed worked, thank you for that workaround. My environment is Windows, PnP.PowerShell 1.5.0.

This thread was closed? The bug is still active. With -interactive or using an Azure AD registered application with full control of sites and read/write of user profiles the error still exists.

Bot closed this issue due to inactivity, have opened it. The best option now would be to contact MS support and explain the case to them.

Right. Which they will point back to this thread as its a community driven module and not maintained by Microsoft. I'll probably fall back to the old Sharepoint module and hope PnP gets updated as a work around.

You can do a minimal CSOM code and use the same access token for the support case. The underlying code is all CSOM C#. Unfortunately we have no clue if and when it will be fixed at all. Other cmdlets work perfectly fine with this auth flow but somehow this particular one doesn’t.

@wimp777 and as mentioned above the workaround for Azure AD registered application is to use Granting access using SharePoint App-Only, which works if you also add the scope ** URL to those instructions. I know it's not the same as Azure AD registered but it's close and it works.

As @erwinvanhunen mentioned earlier in this thread, this isn't an issue with PowerShell PnP but dependant on MSFT's actions.

Please share your case number so I can look into this. Either here or send it to me privately.

25392520

Avel Pinales
Senior Consultant
P 216.359.7866 | E ***@***.******@***.***>

***@***.***D761D1.DF053AC0]
Sent: Tuesday, June 15, 2021 10:29 AM
To: pnp/powershell ***@***.***>
Cc: Pinales, Avel ; Comment ***@***.***>
Subject: Re: [pnp/powershell] [BUG] Set-PnPUserProfileProperty : Access denied. You do not have permission to perform (#277)

@veronicageek< URL I opened a case with Microsoft and escalated this with their SharePoint Online engineers for this issue. I was told that this is not an issue on their end and that the issue is with the module itself.

Please share your case number so I can look into this. Either here or send it to me privately.

-
You are receiving this because you commented.
Reply to this email directly, view it on GitHub< URL or unsubscribe< URL

________________________________
Important Notice: This email message and any files transmitted with it are confidential and intended solely for the use of the individual or entity to whom they are addressed. If you are not the named addressee, you should not disseminate, distribute or copy this e-mail. Please notify the sender immediately by e-mail if you have received this e-mail by mistake and delete this e-mail from your system. Please note that any views or opinions presented in this email are solely those of the author and do not necessarily represent those of Core BTS. Core BTS specifically disclaims liability for any damage caused by any virus transmitted by this email.

Thanks for sharing @apinales. Looking into it with the support team.

@apinales Discussed this with the support team. We would very much like to dive into this issue but since your case has been closed two months ago already we would like to ask you to open a new case and have it reference your old case. If you can share the new case number with me, we'll ensure the correct details will be transferred to the engineer who will handle the case.

Thanks for creating the ticket @apinales. I'm already aware of the ticket number, no need to share.

I am running into this issue right now. Some updates:

- It was working fine with SharePointPnPPowerShellOnline fine, issue started appearing when we migrated to PnP.PowerShell 1.5.0.
- Raised a case with Microsoft support and they are looking into it.
- Got some updates from them to upgrade to 1.6.0 to see if that resolves the issue. However, it does not. Facing same issue with PnP.PowerShell version 1.6.0
- Tried -UseWebLogin and it does work. -Interactive and -Credentials do not work. However, I can't really use -UseWebLogin since we are running this script from an Azure automation account runbook.
- Re-ran Register-PnPManagementShellAccess and provided global admin consent again, that didn't change outcome.
- I don't get "Access Denied" error text. For me, the error message is : "This operation requires you to be managing your own data or have administrator privileges." I tried this with SharePoint administrator and even a global administrator role role, however outcome is same.

@mohandivraniya can you share your case number so we can bundle it with the one we already have ongoing at the moment for this?

In my repro I'm getting the exact same error message as you. What stumbles me here is that you say it works with SharePointPnPPowerShellOnline. What version of it were you using? How did you connect using that?

My case number - 26234370
I am using SharePointPnPPowerShellOnline version 3.23.2007.0.
I am connecting using Connect-PnPOnline -Url > -UseWebLogin

@mohandivraniya that explains it then. WebLogin will work as it works through a cookie to authenticate. The issue I believe we are all facing here is that we cannot register an application in AAD, apply the SharePoint Application principal Users.ReadWrite.All and use that to update a user profile. You would either need to assign the same client Id social permissions through AppInv.aspx as well to make it work or alternatively use -UseWebLogin.

Here's a non-interactive method for connecting that does work to change user profile properties from a script running in an Azure runbook or function: Example 3 on this page. You must make one change. Follow this article Granting access using SharePoint App-Only, but add the scope /sharepoint/**social**/tenant. Update the Permission Request XML as follows, and it will permit writing user profile properties.

This ACS method is retired now but _explicitly still supported for SharePoint use_

Thanks @Markarend for taking the time to detail out what I mentioned above to help others better understand the workaround. Still I believe that should be seen as a workaround. We're currently trying to see if filing a hotfix request for this has any chance of succeeding.

It would be awesome if Microsoft could fix this for Azure AD app-only contexts. This has been an issue for a while, and you lose a bit of transparency for your app registrations when you are building AD Apps, and subsequently have to register some permissions though appinv.aspx

We are tripping over this as well. Found this thread and it's awesome that there are workarounds and - more important - this is being worked on. 👍

Standing in line and waiting for the engineers to click "ship it" ;)

Turns out engineering already had a backlog item open to implement those permissions. The need to have this implemented is raised to engineering through a Critical Design Change Request (CDCR) now based on the discussion here with the aim of prioritizing the backlog item on upcoming sprints. They're currently looking into the efforts it would take to implement this.

Just as an update, the topic is still on the table and has not been forgotten. No decision has been made yet if this functionality will be adjusted or not unfortunately.

Any update from Microsoft SharePoint Online on this issue?????

Its still under discussion. Vacation period makes things progress slow unfortunately. Also because there is an easy workaround available, it takes more effort to raise the priority on this.

@KoenZomers Thanks for updating me. The easy workaround you are talking about is app-only credential?

The easy workaround is to apply what is written here:
 URL

So you can combine your AAD App Registration with ACS permissions to make this work. You would just need to get another token from ACS to handle the user profile stuff, if you directly communicate with the SharePoint/Graph APIs. If you use PnP PowerShell, this will be done automatically for you and you just need to set the permissions as mentioned in the referenced post.

@KoenZomers Thanks, mate. I give it a go. Really really hope Microsoft fix it properly so.

I'd like to add that this is something I'd like to see resolved for a current client. While I can (barely) keep up with the subtle nuance and intricacies of things like this, clients/customers often times aren't paying as close attention. As a result those that have to build/deliver/transition solutions that require both Azure App Registration and the legacy ACS permissions have a hard time explaining this to customer teams. It also requires delivery of code that will be known to have a limited life span in that ACS will only likely be supported for some (unknown) period of time. I'd like to vote this up in priority / escalation such that administrators can have custom SPO user profile sync jobs that don't require ACS any longer.

Hi,
I tried the workaround method suggested using SharePoint-App Online registration, and using this function of pnp powershell code : Set-PnPUserProfileProperty
I am now getting error 401, did i miss something ?

@jeffangama did you add an AppPermissionsRequest as described here

When using SharePoint App-Only, must add the scope /sharepoint/social/tenant. Update the Permission Request XML as follows, and it will permit writing user profile properties.

@Markarend thx for you contribution to this topic. Very helpfull ! I was using the SharePoint App-Only method already. It works at our test tenant, but at the customer it gives an error.

I used exactly the same XML as you do at /_layouts/15/appinv.aspx

I used this code:

$GLOBAL:spAppOnlyClientId = "XXXX"
$GLOBAL:spAppOnlyClientSecret = "XXXX"
$GLOBAL:connection = Connect-PnPOnline -url " URL -clientId $GLOBAL:spAppOnlyClientId -clientSecret $GLOBAL:spAppOnlyClientSecret
(Get-PnPUserProfileProperty -Account " " -Connection $GLOBAL:connection).UserProfileProperties

The error I receive is:
`The remote server returned an error: (401) Unauthorized.`

Do you have any clue why this isn't working?

As an alternative I tried the -UseWebLogin method, which works. But it should run in a Azure runbook, so this method is of no use to us.

@MartijnMolegraaf Check if ACS auth is disabled on the tenant where it gives you the unauthorized.

powershell
Connect-SPOService -Url URL
Get-SPOTenant | select DisableCustomAppAuthentication

Ensure it reports back False. If not, you need to set it to false using:

powershell
Set-SPOTenant -DisableCustomAppAuthentication $false

As an update to this item. We have discussed the situation with engineering. They are aware of this situation and this GitHub thread and are evaluating their options.

Morning @KoenZomers . You made my day, that was indeed the cause and solution. Thank you !

In addition, it is appreciated that you have discussed this case and looked at possible options.

Not yet unfortunately. Its not getting priority yet to be looked at. We'll keep pushing.

@KoenZomers - When the problem is fixed from MSFT side, will it be automatically updated in the PnP module or will we have to also fix our code?

@veronicageek depends a bit on how they decide to fix it. Most probably it will not need code changes from our end.

What would currently be the best way to update SharePoint profile photos despite of this bug?

Utilising New-PnPUPABulkImportJob achieves the same result in a roundabout way

@maartendamen have a look at the documentation: URL . The idea is that you write the updates you want to do in a JSON file, upload it somewhere to SharePoint Online and then call this cmdlet to ask to process it for you. Alternatively you can also directly connect Azure Active Directory with the SharePoint Online User Profile store using URL

Permissions wise you will not solve anything with this by the way.

Is there any way to upvote this issue at Microsoft? Because the old module SharePointPnPPowerShellOnline still works for me. But it seems that it will be retired. And the new method doesn't work. That doesn't feel right for a production environment.

I know this one has been open for quite some time already and that there are a lot of organizations that are hitting this issue. I can say that we're closer than ever on getting this addressed. I'm not at liberty to give any timelines on it, but know that we are still actively working on it internally to get it resolved. Thank you for your patience everyone.

No need to. It has priority already.

You're okay with using the SharePointPnPPowerShellOnline library for now. If it works, it works. It will likely also still work for you with PnP.PowerShell as long as you use the same login method, i.e. -UseWeblogin. Nothing has changed with regards to that.

Everyone - I'm working at Microsoft Support, and with @KoenZomers and Engineering on this issue.
Microsoft Engineering has analyzed this issue and is currently working on a fix. It requires collaboration between multiple teams and there is no ETA for the moment. I will update this thread as updates occur

I tested using SharePointPnPPowerShellOnline legacy module and it works at my end, although it is older module than PnP.PowerShell.

Still patiently waiting on Microsoft to fix this (not the community).

Still not working with
ConnectPnPOnline -Url "myco-admin.sharepoint.com" -ClientId "ZZZZZZ" -Tenant "myco.onmicrosoft.com" -Thumbprint "XXXXXXX"

When will MSFT fix this ?

Set-PnPUserProfileProperty : This operation requires you to be managing your own data or have administrator privileges.
At line:12 char:5
+ Set-PnPUserProfileProperty -Account xxxxxx -Prope ...
+ ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
 + CategoryInfo : WriteError: (:) [Set-PnPUserProfileProperty], ServerException
 + FullyQualifiedErrorId : EXCEPTION,PnP.PowerShell.Commands.UserProfiles.SetUserProfileProperty

Everyone - I checked with Engineering a few day ago, which confirmed they are currently working on a hotfix for this issue.
The plan is to release it this calendar year, I don't have a more specific date.

I had success using this command "Connect-PnPOnline -s -url $Url -Credentials $Creds"

We had the same. We call it a work around, becasue we want to quit using a credential. Our line is.
Connect-PnPOnline -Url $AdminSiteURL -Credentials $credential -SPOManagementShell

@WiljanD I got no update since my last message but I just asked for one, I'll post back here if/when I get an answer

Time is just about up on this. Can we expect this to be fixed anytime soon? I have been patiently waiting for almost 2 years.

@Yvand Hello, have you got any news/ETA ?
For anyone having issue with non interactive login please refer to @Markarend comments which have a workaround while waiting for the Microsoft fix, especially this one URL

@kodsu yes, I got confirmation from Engineering that they are actively working on this and they already completed a good part of the work, although it is not finished yet.
They did not communicate a timeline when it will ship in production, but it will not be before CY 2023.

2023 is here. hope the teams working on this will get us a solution soon.

Today I got an update from Engineering: They continued the work and made progress since my last message, but they still don't have a more precise timeline to communicate, very unfortunately

Imagine being a PM and getting this response from engineering ⛈😉

@heinrich-ulbricht I'm just the messenger who understands the frustration of people in this thread who are waiting for this fix for so long.
I take the time to chase Engineering and provide whatever update I have, even if it is a poor one.
I could have chosen to just remain silent

Is the current work-around still working?

I created a new Azure AD App + secret.
Next on the tenant-admin.sharepoint.com/_layouts/15/appinv.aspx added the FullControl rights to both the content and social scope.
Using Connect-PnPOnline with ClientId and ClientSecret, the Get-PnPUserProfileProperty command is working fine.
But Set-PnPUserProfileProperty returns the following error:

Set-PnPUserProfileProperty : De huidige gebruiker is niet gemachtigd om deze bewerking uit te voeren.
At line:1 char:1
+ Set-PnPUserProfileProperty -Account $acc.AccountName -PropertyName 'S ...
+ ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
 + CategoryInfo : InvalidOperation: (:) [Set-PnPUserProfileProperty], PSInvalidOperationException
 + FullyQualifiedErrorId : InvalidOperation,PnP.PowerShell.Commands.UserProfiles.SetUserProfileProperty

The error text in this Dutch tenant translates the same but the CategoryInfo differs from the topic start.

Did you run the following on each Geo where you need to perform the Get/Set? Also, if multi-geo, you will need to so the ACS registration (AppInv.aspx) on each central admin site.

set-spotenant -DisableCustomAppAuthentication $false

 URL

Found the issue... the error was kind of misleading:
I tried to set SPS-Location but the location value was not yet available in the term store.
Since I first tried Set-PnPUserProfileProperty using certificate authentication instead of using secrets I figured it was still the same issue, but no.

Yes, for sync scenarios, I usually let those termsets remain open - following the model that OOB SharePoint uses for fields like Job Title, etc.

ERROR: This operation requires you to be managing your own data or have administrator privileges.

Getting the same error here when using Azure Function and App Registration.

Is there a fix?

Connect-PnPOnline -Url $URL -UseWebLogin. This is not ideal nor what we want for automation purposes, but it will work. When working on multiple users, I add it to the Begin block and do the work in the Process block of the script in order to pipe information to the script. Hope that helps.

Just wanted to mention that New-PnPUPABulkImportJob works fine with an app registration with Certificate and permissions assigned only in AAD with SharePoint "Sites.FullControl.All" and "User.ReadWrite.All" and no permissions assigned in SPO at /_layouts/15/appinv.aspx , tested it this week in 2 different tenants.

Thanks a lot @jackpoz, will definitely try this out. Glad to get rid of this ACS dependance.

This seems to be working for me. I just fixed an automation I had working.

I was able to test New-PnPUPABulkImportJob with Managed Identity in Automation Account in Azure and it works fine that way too

Good to hear the feedback here that it has started to work for some of you. Just for awareness, the PG is still actively working on completing implementing this scenario. This means for now that it might already work for you, but specific scenarios, such as when you're using taxonomy driven user profile fields, might not work yet. Keep this in mind when switching to the Azure AD / Entra ID option. Test everything thoroughly. If it works for your use case, you're good to go, if it doesn't yet, bear a little longer with us, it's still work in progress.

I got the error: the current user has insufficient permissions to perform this operation for:
SetMultiValuedProfileProperty
SetSingleValueProfileProperty for property: SPS-Location

 However, SetSingleValueProfileProperty does work for other custom properties

I use PNP.framework, set ClientContext using new AuthenticationManager().GetACSAppOnlyContext(siteurl, clientid, secret)

 Thanks

Get-PnPUserProfileProperty works with -ClientSecret but not working with -Thumbprint. Getting Access Denied.

I fixed the issue with the latest PowerShell module on Powershell 7.2 with the command "Sync-PnPSharePointUserProfilesFromAzureActiveDirectory" all information is well documented: URL

The service principal needs to have the following rights:
Graph API:
[User.Read.All]

Sharepoint API
[User.Read.All]
[Sites.FullControl.All]

Which will look like:
 " -AppRole "User.Read.All" -BuiltInType MicrosoftGraph`
`Add-PnPAzureADServicePrincipalAppRole -Principal " " -AppRole "Sites.FullControl.All" -BuiltInType SharePointOnline`
`Add-PnPAzureADServicePrincipalAppRole -Principal " " -AppRole "User.Read.All" -BuiltInType SharePointOnline`

Then also grant the permissions in Sharepoint:
App permission:
`
`
I suspect that this rights package will help in granting rights also for Set-PnPUserProfileProperty, but I started using the latest module because it adds support for using System managed identities and User Managed Identities which I'm using in combination with a automation account to sync the attributes from Azure AD to SharePoint.

The script looks like this:

Connect-PnPOnline tentantname-admin.sharepoint.com -ManagedIdentity -UserAssignedManagedIdentityClientId "Service principal ObjectID"

$users = Get-PnPAzureADUser -select "OnPremisesSamAccountName"

Sync-PnPSharePointUserProfilesFromAzureActiveDirectory -UserProfilePropertyMapping @{"UserName"="OnPremisesSamAccountName";"CellPhone"="MobilePhone"} -Users $users -Folder "Documents"

Disconnect-PnPOnline

Is there a way to assign these permissions using the PnP commandlets? Or some non SharePoint registration?

In my testing the issue with ACL is now solved, and so legacy auth using appreg.aspx is no longer needed.
I have tested using both system managed identity and app reg with cert. Both works fine updating user profile properties including term set fields with only API permissions

`Set-PnPUserProfileProperty -Account adelev@contoso.onmicrosoft.com -PropertyName "SPS-Location" -Value "Stockholm"`

You can ignore the exchange perm as it was only used for testing
image

Looks like TermStore permission is all that was missing. Rest of the commands seem to work now.

Thanks @AndersRask for sharing and @ttd-kevinclement for confirming. I will update our documentation with this great info.

I guess with that, we can also finally close this long running open issue. Thanks for providing all the feedback and thoughts here everyone.

Hi,

If I'm not mistaken the original issue is still not fixed?

I see something was solved concerning ACL, but the original issue was **without using ACL** and since it is going away I'm not going to use it.

So I'm now in the same situation as the OP.

Trying to read certain extension attributes from EntraID profiles and write the values into SharePoint User Profile custom properties.

Here's what I have tried:

# Connect to Microsoft Graph API
Connect-MgGraph -Scopes "User.Read"

# Fetch users where Company Name is "CompanyXYZ"
$users = Get-MgUser -Filter "companyName eq 'CompanyXYZ'"

# Connect to SharePoint Online
Connect-PnPOnline -Url " URL -ClientId

# Update SharePoint user profiles
foreach ($user in $users) {
    $customAttribute = $user.companyName
    Set-PnPUserProfileProperty -Account $user.UserPrincipalName -PropertyName "CompanyName" -Value $customAttribute
}

Set-PnPUserProfileProperty gives error: "This operation requires you to be managing your own data or have administrator privileges."

I have set these (Delegated) API permissions to PnP Powershell and I'm a Global Admin:

Microsoft Graph: User.Read.All
SharePoint: AllSites.FullControl, TermStore.ReadWrite.All, User.ReadWrite.All

Thanks for you suggestion, but I don't think that would help me since eventually I would run this with Managed Identity from Azure Runbook. :(


Question: Issues when running on fly genome?

Dear @biochristmas

The correct command is `ulimit -n`, while you mentioned you used `ulimit -c`. Could you try -n option and see if it helps?

Best
Andrey

Issues when running on fly genome
Hello Andrey:
I have worked with isoquant output on mouse genome and find it super cool. So now I tried to run isoquant on long read seq from a fly RNA seq data, with the attempt to get de novo transcripts in specific knockout model. But it seems to be not working.
My line was:
isoquant.py \
--reference/homes/01/dmmccormick/oliviaao/Genome_assembly_Release_6_plus_ISO1_MT/GCF_000001215.4_Release_6_plus_ISO1_MT_genomic.fna \
--bam METTL3_KO_LacZ_RNAi.bc_3_sorted.bam \
--data_type nanopore \
-o METTL3_KO_LacZ \

Below is the error I encountered:
2025-06-11 14:32:32,590 - CRITICAL - IsoQuant failed with the following error, please, submit this issue to URL (most recent call last):
 File "/homes/01/dmmccormick/miniconda3/envs/isoquant/bin/isoquant.py", line 870, in
 File "/homes/01/dmmccormick/miniconda3/envs/isoquant/bin/isoquant.py", line 864, in main
 File "/homes/01/dmmccormick/miniconda3/envs/isoquant/bin/isoquant.py", line 814, in run_pipeline
 File "/homes/01/dmmccormick/miniconda3/envs/isoquant/share/isoquant-3.7.0-0/src/dataset_processor.py", line 432, in process_all_samples
 File "/homes/01/dmmccormick/miniconda3/envs/isoquant/share/isoquant-3.7.0-0/src/dataset_processor.py", line 456, in process_sample
 File "/homes/01/dmmccormick/miniconda3/envs/isoquant/share/isoquant-3.7.0-0/src/dataset_processor.py", line 547, in collect_reads
 File "/homes/01/dmmccormick/miniconda3/envs/isoquant/share/isoquant-3.7.0-0/src/dataset_processor.py", line 595, in resolve_multimappers
OSError: [Errno 24] Too many open files: 'METTL3_KO_LacZ/OUT/aux/OUT.save_multimappers_NW_001846120.1'

On the other hand, I also tried running isoquant with all the parameters by
--reference /homes/01/dmmccormick/oliviaao/Genome_assembly_Release_6_plus_ISO1_MT/GCF_000001215.4_Release_6_plus_ISO1_MT_genomic.fna \
--genedb /homes/01/dmmccormick/oliviaao/Genome_assembly_Release_6_plus_ISO1_MT/GCF_000001215.4_Release_6_plus_ISO1_MT_genomic.gtf \
--bam METTL3_KO_LacZ_RNAi.bc_3_sorted.bam \
--data_type nanopore \
-o METTL3_KO_LacZ \
This line of code gives me a different error by telling me the GTF file is corrupted. But this GTF file is directly downloaded from NCBI without no modification to anything...?

Please let me know if you could help with these errors, and what I could do to help in this process.
Thank you

Olivia

Dear Olivia

Thank you for the kind feedback!

Regarding the first question - IsoQuant creates a few files for each chromosome in the reference genome. However, every file system has a limit on the number of simultaneously open files.
So I suggest either of two:
- set `ulimit -n `, i.e. `ulimit -n 100000` as discussed here URL
- keep only reference chromosomes in the reference genome (e.g. remove all contigs and patches, or download main assembly file).

Could you give me an error from the second run? It's always possible to set `--no_gtf_check` to skip GTF checking.

Best
Andrey

@AoJR

As the log suggests, tt seems like you annotation has duplicated genes:
` Duplicated gene id Dmel_CG32491 on line 418825`

IsoQuant tried to correct your annotation, but it explicitly wants you to know there is problem. You can try using corrected annotation suggested by IsoQuant:
`An attempt to correct this GTF was made, the result is written to /homes/01/dmmccormick/oliviaao/2025_drs_longreads/2025_0515_All_context_model_Bam_file/S2_LacZ_gtf/GCF_000001215.4_Release_6_plus_ISO1_MT_genomic.corrected.gtf`

Best
Andrey

Thank you Andrey. I run with the corrected gtf genenrated from isoquant and it runs smoothly.

OSError: [Errno 24] Too many open files
Hi,
I've been running isoquant-3.3.1-0 recently and encountered an "OSError: [Errno 24] Too many open files" error when processing the chromosome NW_019678884.1. I've attempted to rerun it three times using the "--resume" parameter, but the same error persists. Is there any way to resolve this issue?
微信图片_20231223183858

Dear @shizhuoxing

I think it's an operating system limit, which can be changed using

ulimit -n

I presume setting something like 10000 should be enough.

Best
Andrey

Hi,
I have recently encountered this problem as well. I set it to have no size limit using ‘ulimit -c unlimited,’ but I still receive errors.
erro

I was able to run smoothly after setting the parameter 'ulimit -n 1000000', thanks!


Question: [FP]: Wrongly reporting vulnerability CVE-2021-41033 on org.eclipse.osgi-3.18.0?

Yes I think this is good, because even today I see a lot of "unsave http" warnings but no one seems to care!
Making it default redirect seems not to improve the situation here, so I think it should really not be the default.

Sadly the current implementation is only on the transport level otherwise I could think about something like even asking the user the same way as we ask for certificates:

- Skip the mirror / edit the location
- Try https instead
- use http even if its unsecure

At least I think (depending on the setting) P2 might want to filter out unsecure mirrors and somehow notify them that they have to upgrade to https, e.g. by getting a list of mirrors that currently use unsafe transports.

For the UI, I think there should be a Warning whenever there is a site added with http and ask the user:

1. Try https instead
2. cancel the operation

It would therefore also be good to ask on start if one wants to convert all http -> https ...

[FP]: Wrongly reporting vulnerability CVE-2021-41033 on org.eclipse.osgi-3.18.0
### Package URl

pkg:maven/org.eclipse.platform/org.eclipse.osgi@3.18.0

### CPE

cpe:2.3:a:eclipse:equinox:*:*:*:*:*:*:*:* versions up to (excluding) 4.21

### CVE

CVE-2021-41033

### ODC Integration

{"label"=>"Maven Plugin"}

### ODC Version

8.3.1

### Description

Per CVE Affected component:
Eclipse Equinox, at least until version 4.21
cpe:2.3:a:eclipse:equinox:*:*:*:*:*:*:*:* versions up to (excluding) 4.21

only 3PP "org.eclipse.osgi-3.18.0.jar" used, but NOT packing/using the vulnerable 3PP component "Eclipse Equinox", even they are NOT packed as indirect dependency in the environment. And this vulnerability is more specific to IDE and the plugin installation of eclipse. But tool is reporting this vulnerability on org.eclipse.osgi-3.18.0.jar, which is wrong.

From Dependency Check tool team, we need confirmation on these false positives. Could you please validate and confirm?

What is the status of this?
I tried to summarize the problem in order to make it more clear.

The Equinox Platform just bundles a bunch of components which can be also individually used and consumed from maven central. The Equinox Platform has a version number but each bundled component has its own version number (and a maven group/artifact id). The latest version of the Equinox Platform is 4.30, see here for a list of contained components are their version.

CVE-2021-41033 states that all Equinox versions
 org.eclipse.platform
 org.eclipse.osgi
 3.18.0

For an outsider it is hard to determine if your used component suffers from a CVE which just states which Equinox Platform version is affected. In this specific case i think that the OSGi Framework was not vulnerable because of CVE-2021-41033.
The problem was in P2, according to this issue.

However i think this a general problem and lets assume that the OSGi Framework was affected.
CVE-2021-41033 was fixed in Equinox Platform >= 4.21, which contains Equinox OSGi Framework in version 3.17, see here.
So using Equinox OSGi Framework in version 3.17 and above should be fine, but OWASP complains because it mixes up the Equinox Platform version with the Equinox OSGi Framework version.

I think this is general problem and might affect other parts of the Equinox Platform as well like (Equinox CM, Equinox Console, etc..) and it would be nice if we could work out a solution for this.

The eclipse maintainer stated that this is indeed a false positive
and that the mentioned artifact (Equinox OSGi Framework) was never affected by CVE-2021-41033, see here.

So please add this as a false positive, thanks in advance.

CVE-2021-41033: Provide improved support for dealing with http protocol in p2
Improve the handling of http protocol by the p2 framework.

The following links provide background detail:

- URL
- URL

Some work has been done to log warnings when the use of http protocol is detected in some limited contexts:

 URL

Such warnings were considered undesirable in some environments, e.g., inside a corporate firewall, so support was provided to allow logged warnings to be disabled:

 URL

To provide both maximum flexibility and maximum security, I propose to expand the current approach to fully address CVE-2021-41033 and beyond.

I propose that we focus on guarding/monitoring p2's transport layer as a whole such that **all** internet access is taken into consideration, including the following mainstream use cases:
- p2 repository content metadata and artifact metadata, composite children, and repository reference URLs, e.g.,
 - URL
- mirror server URLs, i.e., the `p2.mirrorsURL` property in the artifact metadata, e.g.,
 - URL
- mirror URLs, e.g.,
 - URL
- artifacts URLs, e.g.,
 - URL

I propose three modes of http handling:
1. **allow**
 - This is basically what we have now, i.e., log a warning but allow that warning to be suppressed. The warning will be logged for all http access, not simply once per repository and only for repository URLs.
3. **redirect**
 - Any http URI will be redirected/written to use https such that no http request ever reaches out into the network.
5. **block**
 - Any http URI will be blocked as if the network were down/inaccessible.

One important question is "what should be the default behavior in the absence of an explicit choice of one of those three"? I propose the framework defaults to **block** http access. Product/application builders (or even users) can then choose whether to **allow** such access or **redirect** such access; they will be alerted to that choice by the exception that blocks the access. (Note that within a corporate firewall, it might well be the case that redirection to https causes failures due to missing root certificates so **allow**, while not ideal, may be needed in some limited environments where security is ensured via external mechanisms.)

I have a PR pending to implement the above proposal.

Comments on the proposal are welcome!

Should it not the other way round ;-)

Maven is blocking http for a while and even though it produce some disturbance it seems people can manage this, so **block** as a default seems suitable to me.

Probably better "rewrite" as redirection would need to happen on the server side, this seems similar to the "http-only" setting in firefox: URL

Oops. 😜

I edited the original text of fix the for block!

I personally prefer the term redirect because "rewrite" sounds a little more like we might be modify (i.e., literally re-write) the sources of the URLs when we are actually "directing" the underlying transport layer to use https no matter what. The https-only link doesn't use either term. This is more about style/preference rather than substance, so if someone else feels strongly, I have bigger fish to fry. 😛 In the implementation this is controlled by system properties...

Well on the transport layer it is actually enforce SSL :-D

Maybe it would then be better to not use such naming as allow / block / ... so maybe more make it more generic as its done with cipher suites to have a blacklist of protocols (!), as for example one might also thing that plain ftp is bad, so maybe like this as a default (**block**):

-Dp2.transport.insecure = warn
-Dp2.transport.block = ftp,http

to **allow** but warn:

-Dp2.transport.block =

to allow but do not warn:

-Dp2.transport.insecure = debug
-Dp2.transport.block =

And if one really wants redirect

-Dp2.transport.http = redirect

@akurtakov

Do you guys want the SDK product to specify `-Dp2.httpAction=redirect`?

@jonahgraham

Do you want the EPP products to specify `-Dp2.httpAction=redirect`?

I plan to do this for the Eclipse Installer...

I even think this should be the default value for the parameter if nothing else specified.

One "vague" reason I had for choosing **block** as the default is because this forces people to recognize/notice there is a decision to be made. But I suppose a counter argument is that if 99% of the applications (I made that number up) will want **redirect**, then make that the default. If the SDK, all the EPP packages, and the installer all want to specify **redirect**, it's certainly easier to make that the default.

So if @jonahgraham agrees and no one else objects strongly (with good arguments), I'm happy to change the default...

@iloveeclipse Maybe you have an opinion about the default?

Yes, if the server doesn't support https, there will be an error to that effect. Also, within corporate firewalls, there could be failures because of missing root certificates, which folks sometime avoid by using http, but since most things (touch points) are already specifying https, that's already often an issue.

It's clear I can't make everyone happy all of the time and it's clear that I do not have time to heat up nor cool down oceans. Prompting users about the protocol is just beyond the scope of what I plan to do. Such URLs can first appear within composites, or from references to other sites, i.e., places where the user just doesn't see where the URL comes from nor can the user change it...

I will change the default to minimize the amount of effort needed by product/application builders and I will address some of the concerns in via:

 URL

As such , both -Dp2.ftpRule and -Dp2httpRule are supported and both are `redirect` by default.

I will not move the logic outside of the transport layer because it's way too likely some caller will overlook:
image

Also , the current implementation is hard coded so there is currently no use case. Should such a use case arise in the future, it can be addressed at that time.

You asked for feedback here

So I gave my feedback on where I think things *might* be improved, obviously you are free to _ignore_ them, _adapt_ how you seem best fit or whatever else, so for me its not about making anyone *happy* ...

I think the use-case is already described here (even though that's not necessarily my use case or the eclipse use case), lets say I have **one** site that do not supports https (for whatever reason), this will currently fail unless I allow unsecure access for **every** site.

Your feedback has been helpful and constructive and has been adapted into the implementation to make it better and more general than it would have been without your feedback. 😄

FYI, that was changed during the previous release cycle by this commit:

 URL